00010000:00010000:1.0F:1778626050.017759:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0F:1778626050.785768:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778626030/real 1778626030] req@ffff954358788700 x1865013828344064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626050.785801:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828344064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626050 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626050.785816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626050.785832:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626050.785835:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344064:192.168.204.102@tcp:8:kworker.0 00000100:00080000:3.0F:1778626050.785896:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626050.785906:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626050.785908:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626050.785910:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626050.785911:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626050.785913:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626050.785914:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626050.785916:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626050.785917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626050.785919:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626050.785921:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626050.785924:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10369 00000100:00080000:3.0:1778626050.785932:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1778626050.785937:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626050.785945:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626050.785966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828344704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626050.786014:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344704:192.168.204.102@tcp:8:kworker.0 00010000:00010000:2.0:1778626051.041850:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1778626052.065795:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626053.089907:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626053.857770:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778626033/real 1778626033] req@ffff954358789880 x1865013828344320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626053.857793:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828344320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626053.857800:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778626033/real 1778626033] req@ffff954358788000 x1865013828344448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626053.857807:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828344448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626053.857812:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778626033/real 1778626033] req@ffff954358789180 x1865013828344576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626053.857819:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828344576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626053.857839:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778626033/real 1778626033] req@ffff954358789c00 x1865013828344192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626053.857847:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828344192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626053 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626053.857859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626053.857873:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626053.857875:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344320:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778626053.857878:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626053.857881:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626053.857883:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344448:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778626053.857885:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626053.857888:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626053.857890:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344576:192.168.204.102@tcp:8:kworker.0 00000100:00080000:2.0:1778626053.857893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626053.857895:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626053.857897:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828344192:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626053.857921:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626053.857926:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626053.857929:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626053.857933:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10373 00000100:00080000:3.0:1778626053.857941:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1778626053.857946:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626053.857973:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828344832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626053.857992:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626053.857994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626053.857995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626053.857998:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10373 00000100:00080000:3.0:1778626053.858001:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1778626053.858003:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626053.858010:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626053.858017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828344960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626053.858024:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626053.858026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626053.858027:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626053.858030:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828344832:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626053.858031:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10373 00000100:00080000:3.0:1778626053.858035:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1778626053.858038:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626053.858040:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626053.858046:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828345088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626053.858051:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626053.858053:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626053.858054:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626053.858060:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10373 00000100:00080000:3.0:1778626053.858063:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1778626053.858065:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626053.858069:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626053.858075:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828345216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626053.858081:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626053.858083:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1778626053.858122:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626053.858147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345088:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626053.858165:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345216:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626054.114408:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626055.138184:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0F:1778626055.905777:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626055.905798:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626055.905800:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626055.905802:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626055.905804:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626055.905806:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626055.905807:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626055.905808:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626055.905810:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626055.905811:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1778626056.161838:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626057.185838:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1778626058.209818:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1778626058.913820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626058.913828:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1778626058.913831:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626058.913835:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1778626058.913837:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626058.913840:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1778626058.913842:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626058.913845:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1778626058.913848:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626058.913850:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1778626059.233826:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1778626060.258746:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626061.282491:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1778626062.305830:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626063.330017:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1778626064.162344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626064.162352:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626064.162354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626064.162357:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626064.162358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626064.162360:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626064.162362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626064.162363:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1778626064.162365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626064.162366:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1778626064.353964:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626065.377848:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1778626066.402084:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626067.425996:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1778626068.450054:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1778626069.281783:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626069.281790:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626069.281791:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626069.281794:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626069.281795:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626069.281797:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626069.281798:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626069.281800:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626069.281801:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626069.281802:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1778626069.473846:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1778626070.497914:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1778626071.254288:0:189104:0:(file.c:6227:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff954348373a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1778626071.254296:0:189104:0:(file.c:6122:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff954348373a88),name=/ 00000002:00010000:0.0:1778626071.254312:0:189104:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1778626071.254326:0:189104:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1778626071.254337:0:189104:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1778626071.254343:0:189104:0:(namei.c:950:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1778626071.254348:0:189104:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff954348373a88) 00000080:00010000:0.0:1778626071.254351:0:189104:0:(llite_internal.h:1985:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff954348373a88) for lock 0x56379fd28fe4eead 00000080:00010000:0.0:1778626071.254364:0:189104:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x56379fd28fe4eead from it ffffa9c7c0d17cb8 00010000:00010000:0.0:1778626071.254367:0:189104:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1778626071.254372:0:189104:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:0.0:1778626071.254769:0:189104:0:(file.c:6227:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff954348373a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1778626071.254772:0:189104:0:(file.c:6122:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff954348373a88),name=/ 00000002:00010000:0.0:1778626071.254778:0:189104:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1778626071.254783:0:189104:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1778626071.254788:0:189104:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1778626071.254799:0:189104:0:(namei.c:950:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1778626071.254802:0:189104:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff954348373a88) 00000080:00010000:0.0:1778626071.254803:0:189104:0:(llite_internal.h:1985:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff954348373a88) for lock 0x56379fd28fe4eead 00000080:00010000:0.0:1778626071.254806:0:189104:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x56379fd28fe4eead from it ffffa9c7c0d17cb8 00010000:00010000:0.0:1778626071.254808:0:189104:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1778626071.254812:0:189104:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1778626071.254863:0:189104:0:(file.c:6122:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff954348373a88),name=/ 00000002:00010000:0.0:1778626071.254869:0:189104:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1778626071.254873:0:189104:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1778626071.254878:0:189104:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1778626071.254882:0:189104:0:(namei.c:950:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1778626071.254885:0:189104:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff954348373a88) 00000080:00010000:0.0:1778626071.254886:0:189104:0:(llite_internal.h:1985:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff954348373a88) for lock 0x56379fd28fe4eead 00000080:00010000:0.0:1778626071.254888:0:189104:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x56379fd28fe4eead from it ffffa9c7c0d17bb0 00010000:00010000:0.0:1778626071.254890:0:189104:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1778626071.254894:0:189104:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954358970600/0x56379fd28fe4eead lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xa0dae0fc836048a4 expref: -99 pid: 188454 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1778626071.254905:0:189104:0:(file.c:6594:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff954348373a88), inode mode 41ed mask 44 00000080:00200008:0.0:1778626071.254911:0:189104:0:(file.c:1058:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff954348373a88), flags 100000 00000080:00200080:0.0:1778626071.254977:0:189104:0:(dir.c:1998:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff954348373a88) cmd=c00866a4 arg=7ffcf5845b50 00800000:00000080:0.0:1778626071.254989:0:189104:0:(lmv_obd.c:913:lmv_iocontrol()) lustre-clilmv-ffff9543436d7000: cmd=c00866a4 len=592 karg=ffff9543489da400 uarg=0000000000000000 00800000:00000004:0.0:1778626071.254995:0:189104:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff9543436d7000: age 10320, max_age 10409 00000100:00080000:0.0:1778626071.255030:0:189104:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff954342f4b800 x1865013828345344/t0(0) o41->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626071.255041:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626071.521977:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626072.289789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626072.546206:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626073.313787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626073.570062:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626074.337779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626074.401742:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626074.401748:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626074.401750:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626074.401753:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626074.401755:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626074.401756:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626074.401758:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626074.401760:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626074.401761:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626074.401763:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1778626074.593915:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626075.361738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778626075.361745:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1778626050/real 0] req@ffff95434701ea00 x1865013828344704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626075 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626075.361771:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828344704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626075 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1778626075.617887:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626076.385758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626076.641822:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626077.409751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626077.665837:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626078.433781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778626078.433806:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1778626053/real 0] req@ffff95434701f480 x1865013828344960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626078.433846:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828344960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626078.433857:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1778626053/real 0] req@ffff95434701e680 x1865013828345088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626078.433869:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828345088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626078.433877:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1778626053/real 0] req@ffff95434701ca80 x1865013828345216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626078.433909:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828345216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626078.433917:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1778626053/real 0] req@ffff95434701fb80 x1865013828344832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626078.433928:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828344832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626078 ref 2 fl UnregRPC:XNU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1778626078.689874:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626079.457746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626079.521759:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626079.521766:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1778626079.521802:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626079.521804:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1778626079.521806:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626079.521808:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1778626079.521810:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626079.521811:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1778626079.521812:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626079.521814:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1778626079.713848:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626080.481738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626080.737819:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626081.505737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626081.762033:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626082.529736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626082.786204:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626083.553799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626083.809970:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626084.577743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626084.641813:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626084.641819:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626084.641821:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626084.641824:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626084.641825:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626084.641827:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626084.641828:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626084.641841:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1778626084.641843:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626084.641845:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1778626084.833905:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626085.601768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778626085.665900:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.204.102:988 00000400:00000100:3.0:1778626085.665958:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778626085.665973:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626085.666005:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626085.666011:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626085.666017:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626085.666024:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00080000:0.0:1778626085.666040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666067:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666073:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344960:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778626085.666081:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666087:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666091:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345088:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778626085.666097:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666103:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778626085.666107:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626085.666108:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345216:192.168.204.102@tcp:8:kworker.0 00000100:00080000:2.0:1778626085.666115:0:188908:0:(pinger.c:247:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1778626085.666118:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626085.666119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626085.666123:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626085.666127:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626085.666132:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10393 00000100:00080000:2.0:1778626085.666138:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626085.666144:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626085.666148:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828344832:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778626085.666155:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1778626085.666160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666168:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666173:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828344704:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626085.666188:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828345472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626085.666204:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626085.666207:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626085.666208:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626085.666213:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10393 00000100:00080000:2.0:1778626085.666221:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626085.666223:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626085.666226:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626085.666241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626085.666242:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828345600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626085.666253:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626085.666255:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626085.666257:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626085.666259:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10393 00000100:00080000:2.0:1778626085.666262:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626085.666265:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626085.666268:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626085.666274:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788000 x1865013828345728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626085.666280:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626085.666282:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626085.666283:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626085.666285:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10390 00000100:00080000:2.0:1778626085.666287:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626085.666289:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626085.666291:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626085.666296:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828345856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626085.666304:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626085.666306:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626085.666307:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778626085.666314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626085.666317:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10393 00000100:00080000:2.0:1778626085.666321:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626085.666324:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626085.666333:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828345984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626085.666367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626085/real 1778626085] req@ffff954358789c00 x1865013828345472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626085.666387:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828345472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626085.666396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666399:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666402:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626085.666415:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626085.666435:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626085.666449:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626085/real 1778626085] req@ffff954358789180 x1865013828345600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626085.666460:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828345600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626085.666467:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666470:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666473:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626085.666479:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626085.666499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626085.666509:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626085/real 1778626085] req@ffff954358788000 x1865013828345728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626085.666528:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828345728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626085.666536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666539:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666542:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626085.666548:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626085.666568:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626085.666577:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626085/real 1778626085] req@ffff954358789880 x1865013828345856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626085.666588:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828345856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626085.666595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666599:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666602:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828345856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626085.666607:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828345984:192.168.204.102@tcp:250:kworker.0 00000800:00000100:0.0:1778626085.666624:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626085.666635:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626085/real 1778626085] req@ffff954358788700 x1865013828345984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626085.666645:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828345984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626085.666662:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626085.666666:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626085.666669:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828345984:192.168.204.102@tcp:250:kworker.0 00010000:00010000:1.0:1778626085.857952:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626086.625756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626086.881841:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626087.649783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626087.905849:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626088.673804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626088.929811:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626089.697822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626089.953837:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626090.721763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626090.785777:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626090.785785:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626090.785789:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626090.785797:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10425 00000100:00080000:3.0:1778626090.785814:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626090.785820:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626090.785852:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828346112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626090.785875:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626090.785878:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626090.785879:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626090.785882:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10425 00000100:00080000:3.0:1778626090.785885:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626090.785887:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626090.785909:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626090.785913:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828346112:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626090.785922:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828346240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626090.785931:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626090.785934:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626090.785935:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626090.785938:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10425 00000100:00080000:3.0:1778626090.785941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626090.785944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626090.785948:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626090.785959:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828346368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626090.785969:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626090.785973:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626090.785975:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626090.785979:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10425 00000100:00080000:3.0:1778626090.785984:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626090.785989:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626090.785996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626090.786001:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778626090.786011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828346496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626090.786022:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626090.786025:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778626090.786029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346368:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778626090.786034:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626090.786039:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10425 00000100:00100000:0.0:1778626090.786051:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346496:192.168.204.102@tcp:8:kworker.0 00000100:00080000:3.0:1778626090.786052:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626090.786061:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626090.786066:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626090.786078:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828346624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626090.786110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346624:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626090.977872:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626091.745809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626092.001851:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626092.769757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626093.025813:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626093.793782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1778626093.857986:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.204.102:988 00000400:00000100:0.0:1778626093.858043:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1778626093.858063:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778626093.858101:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778626093.858110:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778626093.858119:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778626093.858126:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778626093.858171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626090/real 1778626093] req@ffff95434701c000 x1865013828346624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626093.858212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828346624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626093.858224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626093.858230:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626093.858234:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346624:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626093.858242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626090/real 1778626093] req@ffff95434701df80 x1865013828346496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626093.858254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828346496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626093.858262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626093.858281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626093.858285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626093.858293:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626090/real 1778626093] req@ffff95434701f800 x1865013828346240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626093.858306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828346240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626093.858313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626093.858316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626093.858320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626093.858327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626090/real 1778626093] req@ffff95434701d880 x1865013828346368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626093.858338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828346368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626093.858346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626093.858349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626093.858352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346368:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626093.858358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626090/real 1778626093] req@ffff95434701c380 x1865013828346112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626093.858369:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828346112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626093.858377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626093.858385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626093.858388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828346112:192.168.204.102@tcp:250:kworker.0 00010000:00010000:1.0:1778626094.049849:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626094.817742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626095.073810:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626095.841787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626095.905789:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626095.905800:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626095.905804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626095.905814:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10430 00000100:00080000:3.0:1778626095.905826:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626095.905833:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626095.905867:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828346752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626095.905896:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626095.905900:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626095.905902:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626095.905907:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10430 00000100:00080000:3.0:1778626095.905911:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626095.905916:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626095.905926:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626095.905939:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828346752:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626095.905943:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828346880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626095.905976:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626095.905979:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626095.905980:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626095.905984:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10430 00000100:00080000:3.0:1778626095.905988:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626095.905991:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626095.905995:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626095.906004:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828347008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626095.906013:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626095.906016:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626095.906018:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626095.906023:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10430 00000100:00080000:3.0:1778626095.906027:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626095.906031:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626095.906036:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626095.906046:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778626095.906052:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828347136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626095.906065:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626095.906069:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626095.906072:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626095.906077:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10430 00000100:00100000:2.0:1778626095.906088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347008:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778626095.906089:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626095.906097:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626095.906105:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626095.906120:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828347264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626095.906129:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626095.906169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347264:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626096.097832:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626096.865773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778626096.929975:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.204.102:988 00000400:00000100:1.0:1778626096.930010:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778626096.930024:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626096.930068:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626096.930078:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626096.930098:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626096.930108:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626096.930115:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626095/real 1778626096] req@ffff95434701c380 x1865013828347136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626096.930144:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828347136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626096.930157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626096.930177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626096.930182:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626096.930190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626095/real 1778626096] req@ffff95434701f100 x1865013828346880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626096.930205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828346880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626096.930211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626096.930214:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626096.930216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828346880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626096.930221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626095/real 1778626096] req@ffff95434701c700 x1865013828347008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626096.930228:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828347008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626096.930232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626096.930235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626096.930236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347008:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626096.930240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626095/real 1778626096] req@ffff95434701d500 x1865013828346752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626096.930247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828346752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626096.930252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626096.930259:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626096.930261:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828346752:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626096.930282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626095/real 1778626096] req@ffff95434701d880 x1865013828347264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626096.930291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828347264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626096.930296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626096.930298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626096.930300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347264:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626097.121816:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626097.889748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626098.146161:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626098.913728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626099.169915:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626099.937764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626100.194136:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626100.961750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626101.025758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626101.025766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626101.025770:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626101.025779:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10435 00000100:00080000:1.0:1778626101.025788:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626101.025794:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626101.025829:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828347392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626101.025855:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626101.025858:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626101.025859:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626101.025862:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10435 00000100:00080000:1.0:1778626101.025866:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626101.025869:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626101.025877:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626101.025887:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828347520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626101.025892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828347392:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626101.025903:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626101.025906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626101.025907:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626101.025910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10435 00000100:00080000:1.0:1778626101.025912:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626101.025915:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626101.025918:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626101.025928:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828347648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626101.025935:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626101.025937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626101.025938:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626101.025946:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10435 00000100:00080000:1.0:1778626101.025950:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626101.025953:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626101.025957:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626101.025965:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828347776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626101.025972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626101.025975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626101.025976:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626101.025980:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10435 00000100:00080000:1.0:1778626101.025982:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626101.025985:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626101.025988:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626101.025999:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778626101.025999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828347904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626101.026044:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347648:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626101.026068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626101.026089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347904:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626101.217824:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626101.985757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626102.242317:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626103.009800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626103.266410:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626104.033749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778626104.097993:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778626104.098040:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778626104.098061:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626104.098097:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626104.098107:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626104.098115:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626104.098136:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626104.098164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626101/real 1778626104] req@ffff954350196680 x1865013828347904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626104.098192:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828347904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626104.098204:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626104.098225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626104.098230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347904:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626104.098239:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626101/real 1778626104] req@ffff954350195f80 x1865013828347520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626104.098254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828347520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626104.098262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626104.098266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626104.098269:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626104.098276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626101/real 1778626104] req@ffff954350197b80 x1865013828347648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626104.098291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828347648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626104.098298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626104.098301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626104.098304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347648:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626104.098311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626101/real 1778626104] req@ffff954350195880 x1865013828347776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626104.098333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828347776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626104.098341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626104.098351:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626104.098355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828347776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626104.098362:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626101/real 1778626104] req@ffff954350194380 x1865013828347392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626104.098374:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828347392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626104.098381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626104.098384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626104.098387:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828347392:192.168.204.102@tcp:250:kworker.0 00010000:00010000:1.0:1778626104.289821:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626105.057762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626105.314231:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626106.081813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626106.145838:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626106.145846:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626106.145851:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626106.145861:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10440 00000100:00080000:1.0:1778626106.145873:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626106.145879:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626106.145911:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828348032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626106.145933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626106.145935:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626106.145937:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626106.145940:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10440 00000100:00080000:1.0:1778626106.145942:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626106.145945:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626106.145953:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626106.145961:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828348160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626106.145968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626106.145971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626106.145972:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626106.145974:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10440 00000100:00080000:1.0:1778626106.145977:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626106.145984:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626106.145988:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626106.145989:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828348032:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626106.146017:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828348288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626106.146025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626106.146027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626106.146028:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626106.146032:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10440 00000100:00080000:1.0:1778626106.146036:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626106.146040:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626106.146045:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626106.146058:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828348416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626106.146074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626106.146078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626106.146079:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626106.146084:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10440 00000800:00000100:0.0:1778626106.146092:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626106.146094:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626106.146101:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626106.146107:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626106.146123:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828348544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626106.146162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626106/real 1778626106] req@ffff954350197800 x1865013828348032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626106.146190:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828348032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626106.146218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626106.146224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626106.146228:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828348032:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626106.146261:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626106.146289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626106.146304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626106/real 1778626106] req@ffff954350195c00 x1865013828348160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626106.146347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828348160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626106.146357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626106.146361:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626106.146364:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626106.146371:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348288:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626106.146396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626106.146407:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626106/real 1778626106] req@ffff954350195500 x1865013828348288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626106.146420:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828348288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626106.146442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626106.146447:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626106.146451:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348288:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626106.146458:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626106.146483:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626106.146495:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626106/real 1778626106] req@ffff954350194a80 x1865013828348416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626106.146508:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828348416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626106.146529:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626106.146534:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626106.146537:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626106.146545:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348544:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626106.146571:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626106.146583:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626106/real 1778626106] req@ffff954350194700 x1865013828348544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626106.146595:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828348544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626106.146604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626106.146608:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626106.146612:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348544:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626106.338023:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626107.105779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626107.362163:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626108.129756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626108.385801:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626109.153770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626109.409796:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626110.177747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626110.433806:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626111.201787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626111.265759:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626111.265769:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626111.265774:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626111.265794:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10445 00000100:00080000:3.0:1778626111.265806:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626111.265812:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626111.265861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828348672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626111.265889:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626111.265893:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626111.265905:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626111.265914:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10445 00000100:00080000:3.0:1778626111.265918:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626111.265922:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626111.265932:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626111.265947:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828348800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626111.265958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828348672:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626111.265986:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626111.265990:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626111.265992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626111.265997:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10445 00000100:00080000:3.0:1778626111.266002:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626111.266006:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626111.266012:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626111.266025:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828348928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626111.266030:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626111.266038:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626111.266040:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626111.266042:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626111.266046:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10445 00000100:00080000:3.0:1778626111.266050:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626111.266053:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626111.266057:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626111.266068:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828349056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626111.266069:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626111/real 1778626111] req@ffff95434701f800 x1865013828348672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626111.266089:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626111.266092:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626111.266093:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626111.266097:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10445 00000100:00100000:0.0:1778626111.266100:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828348672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626111.266107:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626111.266113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778626111.266113:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626111.266121:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:3.0:1778626111.266121:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626111.266128:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828348672:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626111.266133:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828349184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626111.266164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626111.266182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626111.266192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626111/real 1778626111] req@ffff95434701df80 x1865013828348800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626111.266206:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828348800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626111.266214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626111.266218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626111.266222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626111.266228:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348928:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626111.266249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626111.266258:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626111/real 1778626111] req@ffff95434701c000 x1865013828348928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626111.266266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828348928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626111.266272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626111.266274:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626111.266276:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828348928:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626111.266280:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626111.266293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626111.266298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626111/real 1778626111] req@ffff95434701ea00 x1865013828349056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626111.266312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828349056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626111.266317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626111.266319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626111.266321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626111.266325:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349184:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626111.266357:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626111.266365:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626111/real 1778626111] req@ffff95434701fb80 x1865013828349184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626111.266372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828349184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626111.266378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626111.266380:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626111.266382:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349184:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626111.457835:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626112.225733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626112.481811:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626113.249739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626113.505864:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626114.273750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626114.529857:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626115.297738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626115.553811:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626116.321733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626116.385870:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626116.385882:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626116.385887:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626116.385898:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10450 00000100:00080000:3.0:1778626116.385908:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626116.385922:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626116.385958:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828349312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626116.385985:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626116.385988:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626116.385990:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626116.385993:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10450 00000100:00080000:3.0:1778626116.385997:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626116.386000:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626116.386008:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626116.386019:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828349440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626116.386027:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828349312:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626116.386046:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626116.386049:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626116.386052:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626116.386056:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10450 00000100:00080000:3.0:1778626116.386061:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626116.386066:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626116.386072:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626116.386085:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828349568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626116.386106:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626116.386109:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626116.386111:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626116.386115:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10450 00000100:00080000:3.0:1778626116.386121:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626116.386124:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349440:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778626116.386130:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626116.386138:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626116.386152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349568:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778626116.386153:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828349696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626116.386167:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626116.386170:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626116.386172:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626116.386176:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10450 00000100:00100000:0.0:1778626116.386180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349696:192.168.204.102@tcp:8:kworker.0 00000100:00080000:3.0:1778626116.386188:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626116.386193:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626116.386198:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626116.386211:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828349824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626116.386245:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349824:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626116.577799:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626117.345728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626117.601825:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626118.369742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626118.625835:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626119.393729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778626119.457925:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778626119.457970:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778626119.457988:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626119.458029:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626119.458039:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626119.458047:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626119.458057:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626119.458252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626116/real 1778626119] req@ffff95434701fb80 x1865013828349824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626119.458280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828349824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626119.458291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626119.458297:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626119.458300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349824:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626119.458308:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626116/real 1778626119] req@ffff95434701d180 x1865013828349696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626119.458319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828349696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626119.458326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626119.458329:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626119.458332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626119.458338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626116/real 1778626119] req@ffff95434701e680 x1865013828349440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626119.458351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828349440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626119.458359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626119.458362:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626119.458365:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626119.458393:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626116/real 1778626119] req@ffff95434701f480 x1865013828349568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626119.458407:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828349568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626119.458415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626119.458418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626119.458422:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828349568:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626119.458428:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626116/real 1778626119] req@ffff95434701ca80 x1865013828349312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626119.458438:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828349312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626119.458445:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626119.458447:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626119.458450:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828349312:192.168.204.102@tcp:250:kworker.0 00010000:00010000:3.0:1778626119.649821:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626120.417721:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626120.673893:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626121.441736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626121.505811:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626121.505822:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626121.505827:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626121.505838:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10456 00000100:00080000:3.0:1778626121.505851:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626121.505858:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626121.505900:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828349952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626121.505946:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626121.505950:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626121.505952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626121.505957:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10456 00000100:00080000:3.0:1778626121.505961:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626121.505965:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626121.505976:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626121.505994:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828350080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626121.506015:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828349952:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626121.506034:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626121.506038:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626121.506041:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626121.506046:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10456 00000100:00080000:3.0:1778626121.506051:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626121.506067:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626121.506074:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626121.506088:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828350208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626121.506099:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626121.506104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626121.506105:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626121.506109:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626121.506115:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10456 00000100:00080000:3.0:1778626121.506120:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626121.506138:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626121.506144:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626121.506161:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828350336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626121.506170:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626121/real 1778626121] req@ffff95434701ea00 x1865013828349952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626121.506177:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626121.506185:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626121.506189:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626121.506195:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10456 00000100:00080000:3.0:1778626121.506207:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626121.506212:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626121.506216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828349952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778626121.506220:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778626121.506236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778626121.506240:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828350464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626121.506256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626121.506262:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828349952:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626121.506295:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626121.506325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626121.506338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626121/real 1778626121] req@ffff95434701c000 x1865013828350080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626121.506351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828350080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626121.506359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626121.506363:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626121.506366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626121.506378:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350208:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626121.506399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626121.506410:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626121/real 1778626121] req@ffff95434701df80 x1865013828350208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626121.506422:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828350208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626121.506432:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626121.506436:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626121.506439:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350208:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626121.506445:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626121.506477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626121.506487:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626121/real 1778626121] req@ffff95434701f800 x1865013828350336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626121.506499:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828350336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626121.506507:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626121.506511:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626121.506515:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626121.506522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350464:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626121.506546:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626121.506557:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626121/real 1778626121] req@ffff95434701d880 x1865013828350464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626121.506568:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828350464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626121.506577:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626121.506581:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626121.506584:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350464:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626121.697800:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626122.465774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626122.721804:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626123.489749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626123.745833:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626124.513821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626124.769795:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626125.537743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626125.793805:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626126.561738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626126.625791:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626126.625799:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626126.625803:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626126.625810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10461 00000100:00080000:0.0:1778626126.625821:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626126.625843:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626126.625872:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828350592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626126.625898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626126.625900:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626126.625902:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626126.625905:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10461 00000100:00080000:0.0:1778626126.625908:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626126.625910:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626126.625917:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626126.625925:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828350720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626126.625933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626126.625935:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626126.625936:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626126.625938:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10461 00000100:00080000:0.0:1778626126.625941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626126.625943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626126.625946:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626126.625951:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828350592:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626126.625955:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828350848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626126.625964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626126.625966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626126.625972:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626126.625975:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10461 00000100:00080000:0.0:1778626126.625978:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626126.625981:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626126.625985:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626126.625993:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828350976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626126.625999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626126.626001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626126.626003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626126.626005:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10461 00000100:00080000:0.0:1778626126.626008:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626126.626009:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626126.626012:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626126.626022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828351104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626126.626034:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626126.626074:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626126/real 1778626126] req@ffff954342f49880 x1865013828350592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626126.626089:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828350592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626126.626097:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626126.626100:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626126.626110:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828350592:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626126.626116:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626126.626135:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626126.626143:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626126/real 1778626126] req@ffff954342f4bb80 x1865013828350720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626126.626152:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828350720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626126.626157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626126.626159:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626126.626162:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626126.626181:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350848:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626126.626200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626126.626207:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626126/real 1778626126] req@ffff954342f49180 x1865013828350848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626126.626215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828350848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626126.626220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626126.626222:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626126.626224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350848:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626126.626228:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626126.626238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626126.626251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626126/real 1778626126] req@ffff954342f48380 x1865013828350976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626126.626258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828350976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626126.626264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626126.626266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626126.626268:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828350976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626126.626271:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351104:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626126.626282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626126.626287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626126/real 1778626126] req@ffff954342f4aa00 x1865013828351104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626126.626294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828351104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626126.626299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626126.626301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626126.626303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351104:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626126.817816:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626127.585732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626127.841818:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626128.609777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626128.865859:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626129.633773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626129.889827:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626130.657755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626130.913829:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626131.681772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626131.745780:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626131.745788:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626131.745792:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626131.745800:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10466 00000100:00080000:1.0:1778626131.745810:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626131.745815:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626131.745854:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828351232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626131.745874:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626131.745877:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626131.745878:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626131.745881:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10466 00000100:00080000:1.0:1778626131.745884:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626131.745886:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626131.745893:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626131.745901:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828351360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626131.745941:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626131.745943:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626131.745945:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626131.745948:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10466 00000100:00080000:1.0:1778626131.745952:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626131.745954:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626131.745959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828351232:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778626131.745961:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626131.745976:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828351488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626131.745984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626131.745987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626131.745988:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626131.745991:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10466 00000100:00080000:1.0:1778626131.745994:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626131.745997:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626131.746000:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626131.746008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828351616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626131.746014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626131.746017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626131.746018:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778626131.746022:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626131.746025:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10466 00000100:00080000:1.0:1778626131.746035:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626131.746039:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626131.746043:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626131.746053:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828351744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626131.746064:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626131/real 1778626131] req@ffff954350194000 x1865013828351232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626131.746080:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828351232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626131.746088:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626131.746091:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626131.746093:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828351232:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626131.746098:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626131.746113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626131.746121:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626131/real 1778626131] req@ffff954350197100 x1865013828351360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626131.746130:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828351360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626131.746139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626131.746142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626131.746145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626131.746171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351488:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626131.746193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626131.746204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626131/real 1778626131] req@ffff954350196d80 x1865013828351488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626131.746216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828351488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626131.746231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626131.746233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626131.746236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351488:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626131.746240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626131.746256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626131.746262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626131/real 1778626131] req@ffff954350197480 x1865013828351616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626131.746270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828351616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626131.746275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626131.746277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626131.746279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626131.746283:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351744:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626131.746291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626131.746296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626131/real 1778626131] req@ffff954350194700 x1865013828351744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626131.746303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828351744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626131.746308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626131.746309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626131.746316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828351744:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626131.937826:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626132.705760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626132.961911:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626133.729766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626133.985822:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626134.753734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626135.009827:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626135.777763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626136.033808:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626136.801761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626136.865785:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626136.865792:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626136.865796:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626136.865803:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10471 00000100:00080000:2.0:1778626136.865812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626136.865818:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626136.865864:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828351872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626136.865888:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626136.865891:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626136.865892:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626136.865896:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10471 00000100:00080000:2.0:1778626136.865899:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626136.865902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626136.865909:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626136.865922:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828352000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626136.865924:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828351872:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626136.865937:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626136.865955:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626136.865957:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626136.865960:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10471 00000100:00080000:2.0:1778626136.865963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626136.865966:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626136.865971:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626136.865983:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828352128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626136.865994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626136.865997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626136.865999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626136.866004:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10471 00000100:00080000:2.0:1778626136.866009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626136.866014:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626136.866020:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626136.866027:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626136.866034:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828352256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626136.866052:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626136.866056:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352128:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778626136.866057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626136.866060:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626136.866064:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10471 00000100:00080000:2.0:1778626136.866068:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626136.866072:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626136.866081:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352256:192.168.204.102@tcp:8:kworker.0 00000008:00100000:2.0:1778626136.866084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626136.866099:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828352384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626136.866137:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352384:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626137.057844:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626137.825744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626138.082356:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626138.849857:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626139.105957:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626139.873724:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778626139.937986:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778626139.938022:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778626139.938036:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626139.938070:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626139.938079:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626139.938087:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626139.938093:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626139.938131:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626136/real 1778626139] req@ffff954358788e00 x1865013828352384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626139.938158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828352384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626139.938171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626139.938177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626139.938181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352384:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626139.938189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626136/real 1778626139] req@ffff954358789500 x1865013828352256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626139.938200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828352256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626139.938209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626139.938212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626139.938215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626139.938222:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626136/real 1778626139] req@ffff954358788a80 x1865013828352000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626139.938234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828352000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626139.938242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626139.938246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626139.938248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626139.938271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626136/real 1778626139] req@ffff95435878bb80 x1865013828352128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626139.938283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828352128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626139.938290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626139.938294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626139.938296:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352128:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626139.938303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626136/real 1778626139] req@ffff954358788380 x1865013828351872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626139.938313:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828351872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626139.938320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626139.938323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626139.938325:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828351872:192.168.204.102@tcp:250:kworker.0 00010000:00010000:1.0:1778626140.130117:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626140.897764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626141.154030:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626141.921790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626141.985782:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626141.985792:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626141.985797:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626141.985807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10476 00000100:00080000:2.0:1778626141.985818:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626141.985824:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626141.985857:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828352512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626141.985883:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626141.985886:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626141.985887:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626141.985891:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10476 00000100:00080000:2.0:1778626141.985894:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626141.985896:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626141.985904:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626141.985914:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828352640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626141.985939:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626141.985944:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828352512:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626141.985944:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626141.985948:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626141.985957:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10476 00000100:00080000:2.0:1778626141.985961:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626141.985964:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626141.985970:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626141.985985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828352768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626141.985998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626141.986002:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626141.986006:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626141.986011:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10476 00000100:00080000:2.0:1778626141.986014:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626141.986021:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626141.986025:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778626141.986027:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626141.986041:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828352896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626141.986052:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626141.986055:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626141.986057:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626141.986061:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10476 00000100:00080000:2.0:1778626141.986072:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626141.986077:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626141.986083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626141/real 1778626141] req@ffff95435878b100 x1865013828352512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778626141.986085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626141.986104:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828353024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626141.986111:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828352512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626141.986128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626141.986143:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626141.986148:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828352512:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626141.986173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626141.986205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626141.986218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626141/real 1778626141] req@ffff95435878b480 x1865013828352640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626141.986230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828352640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626141.986237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626141.986240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626141.986244:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626141.986249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352768:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626141.986269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626141.986280:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626141/real 1778626141] req@ffff95435878aa00 x1865013828352768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626141.986292:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828352768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626141.986301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626141.986305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626141.986308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352768:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626141.986314:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626141.986348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626141.986359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626141/real 1778626141] req@ffff954358788380 x1865013828352896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626141.986370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828352896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626141.986377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626141.986380:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626141.986383:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828352896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626141.986388:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353024:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626141.986405:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626141.986415:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626141/real 1778626141] req@ffff95435878bb80 x1865013828353024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626141.986426:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828353024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626141.986433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626141.986436:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626141.986439:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353024:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626142.177848:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626142.945783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626143.201911:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626143.969785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626144.225819:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626144.993735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626145.249829:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626146.017791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626146.273860:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626147.041756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626147.105762:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626147.105770:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626147.105774:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626147.105781:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10481 00000100:00080000:2.0:1778626147.105791:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626147.105796:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626147.105837:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828353152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626147.105859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626147.105862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626147.105863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626147.105866:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10481 00000100:00080000:2.0:1778626147.105868:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626147.105871:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626147.105879:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626147.105901:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828353152:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626147.105905:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828353280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626147.105915:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626147.105917:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626147.105918:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626147.105922:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10481 00000100:00080000:2.0:1778626147.105925:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626147.105933:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626147.105937:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626147.105946:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828353408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626147.105952:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626147.105954:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626147.105956:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778626147.105960:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626147.105964:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10481 00000100:00080000:2.0:1778626147.105967:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626147.105969:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626147.105973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626147.105981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828353536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626147.105988:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626147.105991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626147.105992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626147.105997:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626147/real 1778626147] req@ffff954358788a80 x1865013828353152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626147.106003:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10481 00000100:00080000:2.0:1778626147.106009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626147.106013:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626147.106017:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626147.106030:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828353664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626147.106040:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828353152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626147.106053:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626147.106059:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626147.106064:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828353152:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626147.106089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626147.106108:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626147.106117:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626147/real 1778626147] req@ffff954358789500 x1865013828353280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626147.106125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828353280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626147.106131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626147.106133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626147.106136:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626147.106140:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353408:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626147.106150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626147.106156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626147/real 1778626147] req@ffff954358788e00 x1865013828353408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626147.106163:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828353408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626147.106176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626147.106179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626147.106181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353408:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626147.106185:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626147.106199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626147.106204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626147/real 1778626147] req@ffff954358788700 x1865013828353536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626147.106212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828353536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626147.106216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626147.106218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626147.106220:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626147.106224:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353664:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626147.106233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626147.106238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626147/real 1778626147] req@ffff954358789880 x1865013828353664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626147.106245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828353664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626147.106250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626147.106251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626147.106257:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353664:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626147.297806:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626148.065777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626148.321894:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626149.089747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626149.345768:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626150.113779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626150.369859:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626151.137761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626151.393776:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626152.161792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626152.225781:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626152.225796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626152.225802:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626152.225815:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10486 00000100:00080000:1.0:1778626152.225829:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626152.225837:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626152.225880:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828353792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626152.225906:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626152.225909:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626152.225911:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626152.225914:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10486 00000100:00080000:1.0:1778626152.225917:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626152.225920:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626152.225929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626152.225938:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828353920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626152.225945:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626152.225948:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626152.225949:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626152.225952:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10486 00000100:00080000:1.0:1778626152.225959:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626152.225962:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626152.225966:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626152.225984:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828353792:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626152.225987:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828354048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626152.225998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626152.226000:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626152.226001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626152.226005:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10486 00000100:00080000:1.0:1778626152.226009:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626152.226011:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626152.226015:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626152.226025:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828354176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626152.226033:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626152.226035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626152.226036:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626152.226038:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10486 00000100:00080000:1.0:1778626152.226051:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626152.226056:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626152.226061:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626152.226061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626152.226078:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828354304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626152.226110:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626152/real 1778626152] req@ffff954350194a80 x1865013828353792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626152.226127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828353792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626152.226136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626152.226140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626152.226143:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828353792:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626152.226149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626152.226163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626152.226171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626152/real 1778626152] req@ffff954350195500 x1865013828353920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626152.226178:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828353920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626152.226184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626152.226186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626152.226187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828353920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626152.226209:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354048:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626152.226220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626152.226234:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626152/real 1778626152] req@ffff954350195c00 x1865013828354048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626152.226242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828354048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626152.226248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626152.226250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626152.226252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354048:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626152.226256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626152.226268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626152.226273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626152/real 1778626152] req@ffff954350197800 x1865013828354176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626152.226281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828354176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626152.226285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626152.226287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626152.226289:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626152.226292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354304:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626152.226301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626152.226305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626152/real 1778626152] req@ffff954350194380 x1865013828354304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626152.226313:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828354304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626152.226323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626152.226325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626152.226327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354304:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626152.417801:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626153.185758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626153.441878:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626154.209796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626154.465811:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626155.233788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626155.489923:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626156.257778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626156.513836:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626157.281797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626157.345779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626157.345789:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626157.345793:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626157.345800:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10491 00000100:00080000:0.0:1778626157.345810:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626157.345823:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626157.345857:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828354432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626157.345882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626157.345885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626157.345886:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626157.345889:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10491 00000100:00080000:0.0:1778626157.345892:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626157.345894:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626157.345900:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626157.345909:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828354560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626157.345916:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626157.345918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626157.345929:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626157.345932:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10491 00000100:00080000:0.0:1778626157.345938:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626157.345941:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626157.345945:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626157.345952:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828354688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626157.345959:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626157.345962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626157.345963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626157.345967:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10491 00000100:00080000:0.0:1778626157.345968:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626157.345971:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626157.345974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626157.345983:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828354816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626157.345983:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828354432:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626157.346005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626157.346007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626157.346008:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626157.346012:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10491 00000100:00080000:0.0:1778626157.346014:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626157.346017:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626157.346020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626157.346036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828354944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626157.346078:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626157.346118:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626157/real 1778626157] req@ffff954342f48e00 x1865013828354432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626157.346137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828354432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626157.346146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626157.346150:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626157.346154:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828354432:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626157.346174:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626157.346192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626157.346201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626157/real 1778626157] req@ffff954342f4aa00 x1865013828354560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626157.346209:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828354560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626157.346214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626157.346217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626157.346219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626157.346223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354688:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626157.346233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626157.346238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626157/real 1778626157] req@ffff954342f48380 x1865013828354688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626157.346245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828354688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626157.346251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626157.346252:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626157.346254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354688:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626157.346281:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626157.346292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626157.346298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626157/real 1778626157] req@ffff954342f49180 x1865013828354816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626157.346311:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828354816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626157.346316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626157.346319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626157.346321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626157.346324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354944:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626157.346336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626157.346342:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626157/real 1778626157] req@ffff954342f4bb80 x1865013828354944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626157.346350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828354944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626157.346355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626157.346357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626157.346359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828354944:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626157.537786:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626158.305808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626158.561782:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626159.329789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626159.585843:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626160.353798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626160.609871:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626161.377822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626161.633816:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626162.401735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626162.465733:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626162.465741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626162.465745:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626162.465754:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10496 00000100:00080000:1.0:1778626162.465763:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626162.465769:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626162.465797:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828355072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626162.465818:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626162.465821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626162.465822:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626162.465825:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10496 00000100:00080000:1.0:1778626162.465828:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626162.465830:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626162.465837:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626162.465847:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828355200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626162.465855:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626162.465859:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778626162.465861:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828355072:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626162.465868:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626162.465873:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10496 00000100:00080000:1.0:1778626162.465876:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626162.465878:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626162.465882:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626162.465891:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828355328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626162.465898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626162.465901:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626162.465902:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626162.465904:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10496 00000100:00080000:1.0:1778626162.465914:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626162.465916:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626162.465920:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626162.465923:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626162.465931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828355456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626162.465950:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626162.465953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626162.465954:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626162.465957:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10496 00000100:00080000:1.0:1778626162.465960:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626162.465962:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626162.465965:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626162.465967:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626162/real 1778626162] req@ffff954350195880 x1865013828355072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626162.465994:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828355584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626162.466000:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828355072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626162.466010:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626162.466015:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626162.466018:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828355072:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626162.466051:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626162.466077:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626162.466102:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626162/real 1778626162] req@ffff954350197b80 x1865013828355200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626162.466114:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828355200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626162.466128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626162.466130:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626162.466132:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626162.466137:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355328:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626162.466153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626162.466167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626162/real 1778626162] req@ffff954350195f80 x1865013828355328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626162.466175:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828355328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626162.466189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626162.466192:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626162.466194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355328:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626162.466198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626162.466212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626162.466218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626162/real 1778626162] req@ffff954350196680 x1865013828355456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626162.466235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828355456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626162.466241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626162.466243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626162.466245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626162.466249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355584:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626162.466260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626162.466265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626162/real 1778626162] req@ffff954350195180 x1865013828355584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626162.466272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828355584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626162.466277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626162.466279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626162.466285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355584:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626162.657834:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626163.425740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626163.681806:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626164.449737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626164.705858:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626165.473824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626165.729833:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626166.497764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626166.753894:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626167.521756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626167.585798:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626167.585809:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626167.585814:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626167.585824:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10502 00000100:00080000:2.0:1778626167.585836:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626167.585842:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626167.585878:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788000 x1865013828355712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626167.585905:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626167.585908:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626167.585909:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626167.585913:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10502 00000100:00080000:2.0:1778626167.585915:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626167.585918:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626167.585925:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626167.585933:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828355840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626167.585941:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626167.585957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626167.585959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626167.585964:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10502 00000100:00100000:0.0:1778626167.585966:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828355712:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626167.585968:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626167.585973:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626167.585980:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626167.585989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828355968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626167.585997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626167.586000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626167.586003:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626167.586007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10502 00000100:00080000:2.0:1778626167.586012:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626167.586016:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626167.586021:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626167.586034:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828356096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626167.586041:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626167.586049:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626167.586052:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626167.586055:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626167.586060:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10502 00000100:00080000:2.0:1778626167.586064:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626167.586068:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626167.586079:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626167.586084:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626167/real 1778626167] req@ffff954358788000 x1865013828355712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626167.586091:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828356224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626167.586116:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828355712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626167.586129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626167.586134:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626167.586139:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828355712:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626167.586146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626167.586170:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626167.586183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626167/real 1778626167] req@ffff954358789180 x1865013828355840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626167.586195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828355840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626167.586203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626167.586206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626167.586210:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626167.586251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355968:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626167.586283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626167.586296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626167/real 1778626167] req@ffff954358789c00 x1865013828355968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626167.586309:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828355968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626167.586317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626167.586321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626167.586324:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828355968:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626167.586331:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626167.586352:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626167.586363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626167/real 1778626167] req@ffff95435878b800 x1865013828356096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626167.586374:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828356096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626167.586382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626167.586385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626167.586389:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626167.586395:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356224:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626167.586414:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626167.586425:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626167/real 1778626167] req@ffff95435878a300 x1865013828356224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626167.586452:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828356224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626167.586461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626167.586465:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626167.586468:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356224:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626167.777837:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626168.545728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626168.801793:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626169.569741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626169.825882:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626170.593737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626170.849817:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626171.617773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626171.873820:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626172.641744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626172.705770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626172.705779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626172.705784:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626172.705794:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10507 00000100:00080000:1.0:1778626172.705805:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626172.705812:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626172.705849:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828356352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626172.705880:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626172.705883:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626172.705886:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626172.705890:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10507 00000100:00080000:1.0:1778626172.705895:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626172.705900:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626172.705909:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626172.705922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828356352:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626172.705927:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828356480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626172.705962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626172.705966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626172.705969:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626172.705973:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10507 00000100:00080000:1.0:1778626172.705979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626172.705983:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626172.705990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626172.706007:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828356608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626172.706014:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356480:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778626172.706024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626172.706028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626172.706031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626172.706037:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10507 00000100:00080000:1.0:1778626172.706042:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626172.706043:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356608:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778626172.706051:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626172.706057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626172.706083:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828356736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626172.706101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626172.706105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626172.706107:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626172.706112:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10507 00000100:00100000:0.0:1778626172.706136:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356736:192.168.204.102@tcp:8:kworker.0 00000100:00080000:1.0:1778626172.706140:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626172.706148:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626172.706155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626172.706171:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828356864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626172.706205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356864:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626172.897824:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626173.665781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626173.921858:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626174.689751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626174.945763:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626175.713780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778626175.777919:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778626175.777965:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778626175.777982:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626175.778031:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626175.778041:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626175.778049:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626175.778067:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626175.778136:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626172/real 1778626175] req@ffff954350195f80 x1865013828356864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626175.778161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828356864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626175.778172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626175.778178:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626175.778185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356864:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626175.778194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626172/real 1778626175] req@ffff954350196680 x1865013828356736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626175.778229:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828356736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626175.778238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626175.778242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626175.778246:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626175.778252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626172/real 1778626175] req@ffff954350195180 x1865013828356608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626175.778265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828356608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626175.778273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626175.778277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626175.778281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356608:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626175.778287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626172/real 1778626175] req@ffff954350194e00 x1865013828356480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626175.778296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828356480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626175.778301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626175.778304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626175.778306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828356480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626175.778309:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626172/real 1778626175] req@ffff954350196a00 x1865013828356352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626175.778327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828356352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626175.778335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626175.778338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626175.778342:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828356352:192.168.204.102@tcp:250:kworker.0 00010000:00010000:1.0:1778626175.969811:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626176.737788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626176.993843:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626177.761755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626177.825770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626177.825780:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626177.825784:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626177.825794:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10512 00000100:00080000:3.0:1778626177.825805:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626177.825811:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626177.825845:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828356992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626177.825885:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626177.825889:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626177.825892:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626177.825896:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10512 00000100:00080000:3.0:1778626177.825901:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626177.825905:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626177.825914:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626177.825928:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828357120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626177.825940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626177.825944:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626177.825946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626177.825953:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10512 00000100:00100000:0.0:1778626177.825954:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828356992:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626177.825957:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626177.825964:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626177.825970:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626177.825983:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828357248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626177.825995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626177.825999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626177.826001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626177.826005:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10512 00000100:00080000:3.0:1778626177.826034:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626177.826040:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626177.826045:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778626177.826049:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626177.826066:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828357376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626177.826078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626177.826081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626177.826085:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626177.826086:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626177/real 1778626177] req@ffff95434701d500 x1865013828356992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626177.826093:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10512 00000100:00080000:3.0:1778626177.826101:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626177.826107:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626177.826113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626177.826117:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828356992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626177.826130:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778626177.826130:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828357504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626177.826140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626177.826149:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828356992:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626177.826159:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626177.826186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626177.826200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626177/real 1778626177] req@ffff95434701c700 x1865013828357120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626177.826213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828357120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626177.826221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626177.826233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626177.826237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626177.826263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357248:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626177.826286:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626177.826297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626177/real 1778626177] req@ffff95434701f100 x1865013828357248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626177.826309:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828357248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626177.826316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626177.826319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626177.826322:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357248:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626177.826328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626177.826348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626177.826360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626177/real 1778626177] req@ffff95434701c380 x1865013828357376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626177.826371:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828357376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626177.826377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626177.826380:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626177.826383:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626177.826388:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357504:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626177.826417:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626177.826428:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626177/real 1778626177] req@ffff95434701d880 x1865013828357504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626177.826439:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828357504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626177.826448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626177.826451:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626177.826455:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357504:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626178.017811:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626178.785748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626179.041827:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626179.809740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626180.065819:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626180.833742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626181.089803:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626181.857732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626182.113820:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626182.881758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626182.945781:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626182.945790:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626182.945794:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626182.945830:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10517 00000100:00080000:2.0:1778626182.945840:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626182.945845:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626182.945874:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828357632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626182.945909:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626182.945912:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626182.945913:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626182.945930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10517 00000100:00080000:2.0:1778626182.945933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626182.945936:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626182.945944:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626182.945956:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828357632:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626182.945958:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828357760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626182.945969:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626182.945972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626182.945973:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626182.945975:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10517 00000100:00080000:2.0:1778626182.945978:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626182.945981:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626182.945984:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626182.945992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828357888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626182.946009:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626182.946012:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626182.946014:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626182.946017:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10517 00000100:00080000:2.0:1778626182.946020:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626182.946023:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626182.946028:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778626182.946029:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626182.946040:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828358016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626182.946046:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626182.946049:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626182.946050:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626182.946053:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10517 00000100:00080000:2.0:1778626182.946055:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626182.946058:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626182.946062:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626182.946065:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626182/real 1778626182] req@ffff954358789f80 x1865013828357632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626182.946070:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828358144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626182.946095:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828357632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626182.946104:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626182.946108:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626182.946111:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828357632:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626182.946130:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626182.946151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626182.946163:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626182/real 1778626182] req@ffff95435878a680 x1865013828357760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626182.946194:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828357760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626182.946203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626182.946207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626182.946209:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626182.946214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357888:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626182.946230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626182.946238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626182/real 1778626182] req@ffff95435878ad80 x1865013828357888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626182.946251:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828357888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626182.946259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626182.946263:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626182.946266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828357888:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626182.946273:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626182.946293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626182.946303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626182/real 1778626182] req@ffff95435878a300 x1865013828358016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626182.946311:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828358016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626182.946325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626182.946327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626182.946339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626182.946343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358144:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626182.946357:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626182.946363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626182/real 1778626182] req@ffff95435878b800 x1865013828358144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626182.946370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828358144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626182.946375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626182.946377:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626182.946379:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358144:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626183.137787:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626183.905755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626184.161793:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626184.929765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626185.185802:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626185.953727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626186.209791:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626186.977737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626187.233825:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626188.001744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626188.065782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626188.065792:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626188.065797:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626188.065807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10522 00000100:00080000:3.0:1778626188.065829:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626188.065835:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626188.065885:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828358272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626188.065916:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626188.065920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626188.065922:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626188.065927:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10522 00000100:00080000:3.0:1778626188.065931:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626188.065935:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626188.065942:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626188.065954:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828358400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626188.065963:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828358272:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626188.065965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626188.065971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626188.065973:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626188.065976:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10522 00000100:00080000:3.0:1778626188.065978:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626188.065980:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626188.065984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626188.065992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828358528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626188.065999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626188.066002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626188.066003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626188.066011:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10522 00000100:00080000:3.0:1778626188.066015:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626188.066017:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626188.066022:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778626188.066022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626188.066035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828358656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626188.066054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626188.066058:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626188/real 1778626188] req@ffff95434701f800 x1865013828358272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626188.066066:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626188.066071:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626188.066077:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10522 00000100:00100000:0.0:1778626188.066080:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828358272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626188.066083:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626188.066092:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626188.066100:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778626188.066102:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626188.066109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:3.0:1778626188.066113:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828358784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626188.066116:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828358272:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626188.066151:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626188.066171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626188.066180:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626188/real 1778626188] req@ffff95434701df80 x1865013828358400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626188.066189:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828358400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626188.066194:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626188.066196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626188.066199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626188.066203:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358528:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626188.066213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626188.066218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626188/real 1778626188] req@ffff95434701c000 x1865013828358528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626188.066225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828358528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626188.066229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626188.066231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626188.066233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358528:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626188.066236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626188.066245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626188.066250:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626188/real 1778626188] req@ffff95434701ea00 x1865013828358656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626188.066264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828358656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626188.066269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626188.066271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626188.066273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626188.066277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358784:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626188.066288:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626188.066294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626188/real 1778626188] req@ffff95434701ca80 x1865013828358784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626188.066301:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828358784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626188.066306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626188.066307:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626188.066309:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828358784:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626188.257838:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626189.025822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626189.281847:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626190.049821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626190.305909:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626191.073768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626191.329867:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626192.097781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626192.353827:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626193.121759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626193.185807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626193.185814:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626193.185818:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626193.185825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10527 00000100:00080000:1.0:1778626193.185836:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626193.185841:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626193.185872:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828358912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626193.185895:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626193.185898:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626193.185899:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626193.185914:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10527 00000100:00080000:1.0:1778626193.185917:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626193.185921:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626193.185929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626193.185938:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828359040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626193.185947:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828358912:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626193.185949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626193.185956:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626193.185957:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626193.185960:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10527 00000100:00080000:1.0:1778626193.185962:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626193.185965:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626193.185968:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626193.185977:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828359168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626193.185984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626193.185987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626193.185988:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626193.185990:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10527 00000100:00080000:1.0:1778626193.185999:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626193.186002:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626193.186007:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626193.186008:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626193.186017:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828359296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626193.186033:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626193.186036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626193.186037:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626193.186043:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10527 00000100:00100000:0.0:1778626193.186044:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626193/real 1778626193] req@ffff954350197b80 x1865013828358912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626193.186046:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626193.186052:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626193.186058:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626193.186069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828359424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626193.186077:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828358912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626193.186104:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626193.186109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626193.186113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828358912:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626193.186136:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626193.186162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626193.186174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626193/real 1778626193] req@ffff954350195880 x1865013828359040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626193.186183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828359040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626193.186189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626193.186191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626193.186193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626193.186198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359168:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626193.186211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626193.186217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626193/real 1778626193] req@ffff954350194380 x1865013828359168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626193.186224:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828359168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626193.186229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626193.186231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626193.186233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359168:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626193.186243:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626193.186258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626193.186268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626193/real 1778626193] req@ffff954350197800 x1865013828359296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626193.186275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828359296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626193.186281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626193.186283:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626193.186285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626193.186288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359424:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626193.186297:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626193.186302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626193/real 1778626193] req@ffff954350195c00 x1865013828359424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626193.186309:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828359424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626193.186315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626193.186316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626193.186320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359424:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626193.377818:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626194.145763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626194.401858:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626195.169773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626195.425801:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626196.193737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626196.449826:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626197.217778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626197.473893:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626198.241747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626198.305788:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626198.305796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626198.305799:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626198.305807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10532 00000100:00080000:3.0:1778626198.305817:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626198.305838:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626198.305865:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828359552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626198.305891:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626198.305894:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626198.305895:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626198.305898:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10532 00000100:00080000:3.0:1778626198.305901:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626198.305904:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626198.305910:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626198.305919:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828359680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626198.305930:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626198.305933:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626198.305935:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626198.305939:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10532 00000100:00080000:3.0:1778626198.305943:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626198.305948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626198.305953:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626198.305960:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828359552:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626198.305967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828359808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626198.305979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626198.305983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626198.305990:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626198.305993:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10532 00000100:00080000:3.0:1778626198.305998:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626198.306001:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626198.306005:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626198.306015:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828359936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626198.306022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626198.306028:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626198.306028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626198.306029:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626198.306035:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10532 00000100:00080000:3.0:1778626198.306038:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626198.306040:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626198.306044:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626198.306053:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828360064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626198.306075:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626198/real 1778626198] req@ffff95434701f480 x1865013828359552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626198.306099:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828359552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626198.306107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626198.306110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626198.306113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828359552:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626198.306119:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626198.306136:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626198.306144:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626198/real 1778626198] req@ffff95434701e680 x1865013828359680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626198.306152:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828359680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626198.306157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626198.306159:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626198.306161:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626198.306177:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359808:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626198.306187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626198.306192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626198/real 1778626198] req@ffff95434701d180 x1865013828359808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626198.306199:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828359808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626198.306213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626198.306216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626198.306218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359808:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626198.306222:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626198.306235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626198.306241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626198/real 1778626198] req@ffff95434701fb80 x1865013828359936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626198.306248:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828359936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626198.306253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626198.306254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626198.306256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828359936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626198.306260:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360064:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626198.306268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626198.306274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626198/real 1778626198] req@ffff95434701ca80 x1865013828360064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626198.306281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828360064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626198.306286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626198.306287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626198.306294:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360064:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626198.497849:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626199.265751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626199.521852:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626200.289741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626200.545828:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626201.313787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626201.569910:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626202.337769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626202.593826:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626203.361757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626203.425766:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626203.425774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626203.425778:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626203.425786:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10537 00000100:00080000:1.0:1778626203.425796:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626203.425801:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626203.425833:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828360192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626203.425855:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626203.425858:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626203.425873:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626203.425876:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10537 00000100:00080000:1.0:1778626203.425880:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626203.425883:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626203.425891:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626203.425901:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828360320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626203.425908:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828360192:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626203.425924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626203.425926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626203.425927:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626203.425931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10537 00000100:00080000:1.0:1778626203.425933:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626203.425935:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626203.425938:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626203.425947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828360448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626203.425953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626203.425955:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626203.425957:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626203.425959:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10537 00000100:00080000:1.0:1778626203.425967:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626203.425970:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626203.425973:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626203.425978:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626203.425987:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828360576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626203.425994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626203.425996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626203.425998:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626203.426000:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10537 00000100:00080000:1.0:1778626203.426004:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626203.426006:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626203.426009:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626203.426016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828360704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626203.426024:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626203/real 1778626203] req@ffff954350195500 x1865013828360192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626203.426046:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828360192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626203.426057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626203.426061:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626203.426066:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828360192:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626203.426089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626203.426111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626203.426124:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626203/real 1778626203] req@ffff954350194a80 x1865013828360320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626203.426137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828360320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626203.426147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626203.426150:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626203.426154:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626203.426160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360448:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626203.426178:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626203.426197:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626203/real 1778626203] req@ffff954350194700 x1865013828360448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626203.426211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828360448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626203.426219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626203.426223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626203.426226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360448:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626203.426233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626203.426252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626203.426262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626203/real 1778626203] req@ffff954350197480 x1865013828360576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626203.426275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828360576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626203.426283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626203.426286:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626203.426290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626203.426296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360704:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626203.426313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626203.426322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626203/real 1778626203] req@ffff954350196d80 x1865013828360704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626203.426335:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828360704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626203.426349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626203.426353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626203.426356:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360704:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626203.617839:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626204.385774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626204.642723:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626205.409756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626205.666110:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626206.433722:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626206.690075:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626207.457734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626207.714284:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626208.481811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626208.545775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626208.545783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626208.545794:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626208.545802:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10543 00000100:00080000:1.0:1778626208.545812:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626208.545817:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626208.545849:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828360832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626208.545871:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626208.545874:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626208.545876:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626208.545893:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10543 00000100:00080000:1.0:1778626208.545897:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626208.545900:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626208.545909:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626208.545920:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828360960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626208.545934:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626208.545936:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778626208.545938:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828360832:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626208.545938:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626208.545942:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10543 00000100:00080000:1.0:1778626208.545952:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626208.545956:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626208.545959:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626208.545969:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828361088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626208.545976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626208.545979:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626208.545980:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626208.545983:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10543 00000100:00080000:1.0:1778626208.545987:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626208.545989:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626208.545994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626208.546003:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828361216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626208.546013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626208.546016:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626208.546017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626208.546020:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10543 00000100:00080000:1.0:1778626208.546023:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626208.546025:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626208.546030:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626208.546032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626208.546044:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828361344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626208.546090:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626208/real 1778626208] req@ffff954350197100 x1865013828360832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626208.546107:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828360832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626208.546117:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626208.546121:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626208.546124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828360832:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626208.546147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626208.546166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626208.546178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626208/real 1778626208] req@ffff954350194000 x1865013828360960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626208.546191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828360960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626208.546201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626208.546205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626208.546209:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828360960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626208.546230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361088:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626208.546247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626208.546255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626208/real 1778626208] req@ffff954350196d80 x1865013828361088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626208.546264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828361088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626208.546269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626208.546272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626208.546274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361088:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626208.546277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626208.546288:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626208.546294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626208/real 1778626208] req@ffff954350197480 x1865013828361216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626208.546301:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828361216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626208.546306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626208.546308:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626208.546310:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626208.546314:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361344:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626208.546327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626208.546336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626208/real 1778626208] req@ffff954350194700 x1865013828361344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626208.546356:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828361344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626208.546366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626208.546370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626208.546373:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361344:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626208.737896:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626209.505739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626209.761886:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626210.529774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626210.785867:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626211.553731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626211.809793:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626212.577727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626212.833849:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626213.601734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626213.665765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626213.665774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626213.665777:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626213.665785:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10548 00000100:00080000:1.0:1778626213.665795:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626213.665800:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626213.665831:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828361472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626213.665854:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626213.665857:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626213.665858:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626213.665874:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10548 00000100:00080000:1.0:1778626213.665878:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626213.665881:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626213.665889:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626213.665898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828361600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626213.665907:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626213.665912:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828361472:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626213.665912:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626213.665913:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626213.665921:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10548 00000100:00080000:1.0:1778626213.665923:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626213.665925:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626213.665928:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626213.665936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828361728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626213.665943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626213.665945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626213.665946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626213.665949:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10548 00000100:00080000:1.0:1778626213.665963:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626213.665966:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626213.665969:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626213.665974:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626213.665980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828361856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626213.665992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626213.665994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626213.665996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626213.665999:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10548 00000100:00080000:1.0:1778626213.666003:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626213.666008:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626213.666009:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626213/real 1778626213] req@ffff954350194a80 x1865013828361472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778626213.666016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626213.666029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828361984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626213.666032:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828361472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626213.666055:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626213.666059:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626213.666062:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828361472:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626213.666081:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626213.666095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626213.666102:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626213/real 1778626213] req@ffff954350195500 x1865013828361600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626213.666110:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828361600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626213.666127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626213.666129:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626213.666131:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626213.666135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361728:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626213.666149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626213.666155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626213/real 1778626213] req@ffff954350195c00 x1865013828361728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626213.666162:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828361728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626213.666167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626213.666169:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626213.666171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361728:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626213.666174:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626213.666184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626213.666189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626213/real 1778626213] req@ffff954350197800 x1865013828361856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626213.666195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828361856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626213.666200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626213.666202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626213.666210:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626213.666214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361984:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626213.666227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626213.666232:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626213/real 1778626213] req@ffff954350194380 x1865013828361984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626213.666239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828361984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626213.666244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626213.666246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626213.666248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828361984:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626213.857798:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626214.625782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626214.881798:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626215.649743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626215.905814:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626216.673756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626216.929853:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626217.697732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626217.953848:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626218.721745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626218.785757:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626218.785765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626218.785769:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626218.785777:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10553 00000100:00080000:1.0:1778626218.785787:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626218.785793:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626218.785824:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828362112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626218.785846:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626218.785849:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626218.785850:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626218.785853:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10553 00000100:00080000:1.0:1778626218.785865:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626218.785870:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626218.785878:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626218.785888:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828362240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626218.785896:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828362112:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626218.785898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626218.785904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626218.785906:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626218.785909:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10553 00000100:00080000:1.0:1778626218.785911:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626218.785913:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626218.785916:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626218.785925:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828362368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626218.785931:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626218.785934:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626218.785935:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626218.785938:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10553 00000100:00080000:1.0:1778626218.785948:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626218.785951:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626218.785954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626218.785957:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626218.785965:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828362496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626218.785980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626218.785983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626218.785984:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626218.785987:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10553 00000100:00080000:1.0:1778626218.785991:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626218.785994:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626218.785999:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626218/real 1778626218] req@ffff954350195880 x1865013828362112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778626218.786000:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626218.786013:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828362624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626218.786028:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828362112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626218.786038:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626218.786042:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626218.786045:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828362112:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626218.786062:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626218.786088:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626218.786105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626218/real 1778626218] req@ffff954350197b80 x1865013828362240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626218.786113:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828362240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626218.786119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626218.786121:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626218.786124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626218.786127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362368:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626218.786139:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626218.786145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626218/real 1778626218] req@ffff954350196a00 x1865013828362368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626218.786155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828362368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626218.786163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626218.786166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626218.786169:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362368:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626218.786175:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626218.786196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626218.786206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626218/real 1778626218] req@ffff954350194e00 x1865013828362496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626218.786231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828362496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626218.786242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626218.786245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626218.786247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626218.786251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362624:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626218.786267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626218.786274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626218/real 1778626218] req@ffff954350195180 x1865013828362624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626218.786281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828362624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626218.786287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626218.786289:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626218.786291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362624:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626218.977811:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626219.745735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626220.001814:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626220.769772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626221.025941:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626221.793800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626222.049838:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626222.817765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626223.073849:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626223.841796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626223.905808:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626223.905817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626223.905821:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626223.905828:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10558 00000100:00080000:0.0:1778626223.905846:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626223.905851:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626223.905881:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828362752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626223.905903:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626223.905929:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626223.905931:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626223.905935:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10558 00000100:00080000:0.0:1778626223.905939:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626223.905942:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626223.905950:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626223.905960:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828362752:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626223.905961:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828362880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626223.905974:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626223.905976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626223.905977:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626223.905980:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10558 00000100:00080000:0.0:1778626223.905982:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626223.905984:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626223.905987:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626223.905994:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828363008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626223.906000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626223.906002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626223.906003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626223.906006:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10558 00000100:00080000:0.0:1778626223.906008:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626223.906011:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626223.906019:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626223.906029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828363136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626223.906030:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626223.906039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626223.906041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626223.906043:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626223.906045:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10558 00000100:00080000:0.0:1778626223.906054:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626223.906057:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626223.906060:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626223.906068:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626223/real 1778626223] req@ffff954342f49880 x1865013828362752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626223.906069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828363264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626223.906134:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828362752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626223.906144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626223.906148:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626223.906152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828362752:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626223.906171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626223.906197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626223.906206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626223/real 1778626223] req@ffff954342f48000 x1865013828362880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626223.906214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828362880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626223.906220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626223.906222:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626223.906224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828362880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626223.906228:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363008:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626223.906238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626223.906252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626223/real 1778626223] req@ffff954342f4a300 x1865013828363008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626223.906260:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828363008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626223.906265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626223.906268:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626223.906270:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363008:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626223.906273:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626223.906285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626223.906290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626223/real 1778626223] req@ffff954342f4b480 x1865013828363136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626223.906297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828363136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626223.906302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626223.906304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626223.906305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626223.906309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363264:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626223.906323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626223.906333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626223/real 1778626223] req@ffff954342f49c00 x1865013828363264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626223.906345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828363264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626223.906370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626223.906373:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626223.906377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363264:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626224.097821:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626224.865761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626225.121802:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626225.889766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626226.145801:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626226.913753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626227.169804:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626227.937789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626228.193855:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626228.961730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626229.025839:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626229.025851:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626229.025856:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626229.025869:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10563 00000100:00080000:3.0:1778626229.025884:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626229.025891:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626229.025933:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828363392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626229.025962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626229.025965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626229.025967:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626229.025970:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10563 00000100:00080000:3.0:1778626229.025972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626229.025975:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626229.025984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626229.025994:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828363520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626229.026002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626229.026004:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626229.026017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626229.026035:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828363392:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626229.026041:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10563 00000100:00080000:3.0:1778626229.026049:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626229.026053:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626229.026058:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626229.026069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828363648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626229.026078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626229.026081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626229.026082:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626229.026086:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10563 00000100:00080000:3.0:1778626229.026088:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626229.026091:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626229.026096:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626229.026104:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828363776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626229.026114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626229.026117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626229.026119:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778626229.026129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626229.026132:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10563 00000100:00080000:3.0:1778626229.026139:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626229.026144:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626229.026150:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626229.026166:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828363904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626229.026178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626229/real 1778626229] req@ffff95434701ea00 x1865013828363392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626229.026209:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828363392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626229.026222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626229.026227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626229.026231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828363392:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626229.026241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626229.026267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626229.026280:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626229/real 1778626229] req@ffff95434701c000 x1865013828363520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626229.026293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828363520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626229.026329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626229.026333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626229.026337:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626229.026364:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363648:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626229.026388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626229.026400:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626229/real 1778626229] req@ffff95434701df80 x1865013828363648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626229.026413:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828363648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626229.026421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626229.026426:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626229.026429:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363648:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626229.026436:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626229.026457:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626229.026469:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626229/real 1778626229] req@ffff95434701f800 x1865013828363776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626229.026481:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828363776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626229.026489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626229.026492:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626229.026502:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626229.026510:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363904:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626229.026532:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626229.026543:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626229/real 1778626229] req@ffff95434701d880 x1865013828363904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626229.026556:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828363904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626229.026565:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626229.026568:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626229.026571:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828363904:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626229.217794:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626229.985786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626230.241795:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626231.009763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626231.265802:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626232.033807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626232.289980:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626233.057801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626233.313854:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626234.081813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626234.145820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626234.145831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626234.145837:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626234.145847:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10568 00000100:00080000:3.0:1778626234.145860:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626234.145867:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626234.145924:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828364032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626234.145954:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626234.145958:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626234.145959:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626234.145963:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10568 00000100:00080000:3.0:1778626234.145965:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626234.145968:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626234.145978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626234.145988:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828364160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626234.145996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626234.145999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626234.146000:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626234.146003:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10568 00000100:00080000:3.0:1778626234.146005:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626234.146028:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828364032:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626234.146030:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626234.146039:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626234.146048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828364288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626234.146055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626234.146058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626234.146059:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626234.146062:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10568 00000100:00080000:3.0:1778626234.146065:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626234.146068:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626234.146071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626234.146079:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828364416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626234.146086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626234.146088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626234.146089:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626234.146091:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10568 00000100:00080000:3.0:1778626234.146094:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626234.146096:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626234.146099:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626234.146108:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828364544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626234.146117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626234.146164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626234/real 1778626234] req@ffff95434701c380 x1865013828364032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626234.146190:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828364032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626234.146201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626234.146206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626234.146218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828364032:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626234.146226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626234.146250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626234.146263:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626234/real 1778626234] req@ffff95434701f100 x1865013828364160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626234.146275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828364160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626234.146283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626234.146287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626234.146290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626234.146319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364288:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626234.146341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626234.146353:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626234/real 1778626234] req@ffff95434701c700 x1865013828364288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626234.146366:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828364288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626234.146391:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626234.146396:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626234.146400:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364288:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626234.146407:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626234.146431:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626234.146452:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626234/real 1778626234] req@ffff95434701d500 x1865013828364416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626234.146464:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828364416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626234.146473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626234.146476:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626234.146479:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626234.146485:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364544:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626234.146505:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626234.146514:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626234/real 1778626234] req@ffff95434701d880 x1865013828364544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626234.146526:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828364544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626234.146534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626234.146537:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626234.146540:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364544:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626234.337967:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626235.105755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626235.361904:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626236.129822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626236.385967:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626237.153803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626237.409892:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626238.177796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626238.433925:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626239.201825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626239.265773:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626239.265791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626239.265796:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626239.265859:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10573 00000100:00080000:1.0:1778626239.265874:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626239.265883:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626239.265931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828364672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626239.265966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626239.265970:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626239.265972:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626239.265977:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10573 00000100:00080000:1.0:1778626239.265980:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626239.265985:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626239.265997:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626239.266011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828364800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626239.266018:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828364672:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626239.266030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626239.266034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626239.266035:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626239.266040:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10573 00000100:00080000:1.0:1778626239.266042:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626239.266045:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626239.266050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626239.266063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828364928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626239.266086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626239.266089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626239.266091:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626239.266096:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10573 00000100:00080000:1.0:1778626239.266102:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626239.266106:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626239.266111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626239.266123:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828365056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626239.266132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626239.266136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626239.266138:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626239.266143:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10573 00000100:00080000:1.0:1778626239.266149:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626239.266155:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626239.266162:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626239.266172:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828365184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626239.266180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626239.266238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364928:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626239.266263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626239.266294:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365184:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626239.457884:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626240.225737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626240.481855:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626241.249758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626241.505800:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626242.273733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778626242.337912:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778626242.337969:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778626242.337986:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626242.338026:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626242.338036:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626242.338052:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626242.338067:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778626242.338078:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626239/real 1778626242] req@ffff954350195f80 x1865013828364800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626242.338109:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828364800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626242.338122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626242.338128:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626242.338133:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626242.338141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626239/real 1778626242] req@ffff954350195180 x1865013828364928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626242.338156:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828364928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626242.338164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626242.338167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626242.338171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828364928:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626242.338178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626239/real 1778626242] req@ffff954350194e00 x1865013828365056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626242.338201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828365056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626242.338210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626242.338214:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626242.338217:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626242.338225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626239/real 1778626242] req@ffff954350196680 x1865013828364672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626242.338235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828364672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626242.338243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626242.338246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626242.338249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828364672:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626242.338278:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626239/real 1778626242] req@ffff954350196a00 x1865013828365184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626242.338291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828365184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626242.338299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626242.338302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626242.338305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365184:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626242.529816:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626243.297751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626243.553803:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626244.321740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626244.385798:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626244.385806:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626244.385810:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626244.385817:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10578 00000100:00080000:2.0:1778626244.385826:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626244.385831:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626244.385859:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828365312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626244.385884:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626244.385886:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626244.385888:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626244.385891:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10578 00000100:00080000:2.0:1778626244.385893:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626244.385895:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626244.385903:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626244.385911:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828365440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626244.385918:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626244.385920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626244.385922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626244.385924:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10578 00000100:00080000:2.0:1778626244.385926:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626244.385928:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626244.385938:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828365312:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778626244.385948:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626244.385958:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788000 x1865013828365568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626244.385966:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626244.385968:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626244.385969:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626244.385972:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10578 00000100:00080000:2.0:1778626244.385975:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626244.385978:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626244.385981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626244.385988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828365696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626244.385994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626244.385997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626244.386009:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626244.386010:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10578 00000100:00080000:2.0:1778626244.386013:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626244.386015:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626244.386017:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626244.386022:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626244.386027:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828365824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626244.386065:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626244/real 1778626244] req@ffff954358789c00 x1865013828365312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626244.386084:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828365312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626244.386095:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626244.386100:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626244.386104:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828365312:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626244.386111:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626244.386150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626244.386165:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626244/real 1778626244] req@ffff954358789180 x1865013828365440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626244.386179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828365440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626244.386187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626244.386190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626244.386194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626244.386231:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365568:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626244.386252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626244.386264:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626244/real 1778626244] req@ffff954358788000 x1865013828365568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626244.386276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828365568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626244.386284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626244.386288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626244.386292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365568:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626244.386298:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626244.386317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626244.386326:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626244/real 1778626244] req@ffff954358789880 x1865013828365696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626244.386353:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828365696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626244.386361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626244.386364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626244.386368:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626244.386374:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365824:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626244.386395:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626244.386405:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626244/real 1778626244] req@ffff954358788700 x1865013828365824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626244.386418:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828365824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626244.386426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626244.386429:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626244.386432:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828365824:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626244.577839:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626245.345786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626245.601828:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626246.369733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626246.625790:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626247.393822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626247.649995:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626248.417732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626248.673923:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626249.441778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626249.505786:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626249.505799:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626249.505804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626249.505817:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10584 00000100:00080000:0.0:1778626249.505832:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626249.505839:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626249.505889:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828365952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626249.505957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626249.505961:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626249.505963:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626249.505968:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10584 00000100:00080000:0.0:1778626249.505973:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626249.505978:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626249.505989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626249.506003:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828365952:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626249.506005:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828366080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626249.506019:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626249.506022:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626249.506024:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626249.506028:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10584 00000100:00080000:0.0:1778626249.506032:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626249.506035:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626249.506041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626249.506053:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828366208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626249.506062:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626249.506066:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626249.506068:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626249.506072:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10584 00000800:00000100:2.0:1778626249.506082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626249.506092:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626249.506099:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626249.506105:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626249.506118:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828366336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626249.506124:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626249/real 1778626249] req@ffff954342f49500 x1865013828365952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626249.506134:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626249.506139:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626249.506141:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626249.506146:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828365952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626249.506148:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10584 00000100:00080000:0.0:1778626249.506155:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626249.506161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626249.506162:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778626249.506168:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:0.0:1778626249.506170:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626249.506173:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828365952:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626249.506186:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828366464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626249.506199:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626249.506216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626249.506226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626249/real 1778626249] req@ffff954342f4ad80 x1865013828366080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626249.506235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828366080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626249.506240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626249.506243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626249.506245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626249.506248:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366208:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626249.506271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626249.506277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626249/real 1778626249] req@ffff954342f48a80 x1865013828366208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626249.506285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828366208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626249.506291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626249.506293:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626249.506295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366208:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626249.506299:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626249.506310:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626249.506315:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626249/real 1778626249] req@ffff954342f4b100 x1865013828366336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626249.506322:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828366336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626249.506327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626249.506329:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626249.506330:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626249.506343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366464:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626249.506353:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626249.506358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626249/real 1778626249] req@ffff954342f49f80 x1865013828366464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626249.506376:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828366464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626249.506381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626249.506383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626249.506385:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366464:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626249.697991:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626250.465768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626250.721947:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626251.489854:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626251.745944:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626252.513782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626252.769848:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626253.537781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626253.793897:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626254.561741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626254.625778:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626254.625787:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626254.625791:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626254.625799:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10589 00000100:00080000:2.0:1778626254.625815:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626254.625820:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626254.625854:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828366592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626254.625883:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626254.625888:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626254.625890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626254.625905:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10589 00000100:00080000:2.0:1778626254.625910:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626254.625928:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828366592:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626254.625931:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626254.625940:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626254.625954:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828366720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626254.625962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626254.625965:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626254.625967:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626254.625969:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10589 00000100:00080000:2.0:1778626254.625973:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626254.625975:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626254.625979:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626254.625986:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828366848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626254.625992:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626254.626007:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1778626254.626008:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626254.626009:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626254.626014:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10589 00000100:00080000:2.0:1778626254.626016:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626254.626019:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626254.626022:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626254.626029:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828366976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626254.626035:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626254.626041:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778626254.626044:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626254/real 1778626254] req@ffff954358788e00 x1865013828366592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626254.626044:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626254.626051:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10589 00000100:00080000:2.0:1778626254.626057:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626254.626061:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626254.626065:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626254.626070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828366592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626254.626075:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828367104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626254.626088:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626254.626093:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626254.626097:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828366592:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626254.626125:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626254.626143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626254.626152:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626254/real 1778626254] req@ffff954358789500 x1865013828366720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626254.626160:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828366720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626254.626165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626254.626167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626254.626170:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626254.626173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366848:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626254.626183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626254.626188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626254/real 1778626254] req@ffff954358788a80 x1865013828366848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626254.626195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828366848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626254.626200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626254.626202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626254.626204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366848:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626254.626207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626254.626219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626254.626224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626254/real 1778626254] req@ffff95435878bb80 x1865013828366976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626254.626237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828366976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626254.626242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626254.626244:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626254.626246:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828366976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626254.626249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367104:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626254.626261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626254.626266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626254/real 1778626254] req@ffff954358788380 x1865013828367104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626254.626273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828367104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626254.626289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626254.626290:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626254.626292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367104:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626254.817819:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626255.585755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626255.841830:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626256.609733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626256.865824:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626257.633727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626257.889849:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626258.657748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626258.913796:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626259.681731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626259.745760:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626259.745767:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626259.745771:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626259.745780:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10594 00000100:00080000:0.0:1778626259.745789:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626259.745795:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626259.745825:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828367232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626259.745861:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626259.745864:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626259.745866:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626259.745869:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10594 00000100:00080000:0.0:1778626259.745872:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626259.745876:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626259.745885:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626259.745897:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828367360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626259.745897:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828367232:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626259.745911:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626259.745913:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626259.745915:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626259.745917:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10594 00000100:00080000:0.0:1778626259.745920:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626259.745922:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626259.745925:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626259.745935:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828367488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626259.745941:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626259.745943:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626259.745945:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626259.745948:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10594 00000100:00080000:0.0:1778626259.745957:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778626259.745960:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626259.745963:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626259.745968:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626259.745983:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828367616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626259.745991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626259.745993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626259.745995:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626259.745998:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10594 00000100:00080000:0.0:1778626259.746000:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626259.746004:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626259.746007:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626259/real 1778626259] req@ffff954342f48700 x1865013828367232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778626259.746009:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626259.746022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828367744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626259.746030:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828367232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626259.746047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626259.746050:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626259.746054:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828367232:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626259.746073:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626259.746089:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626259.746112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626259/real 1778626259] req@ffff954342f4a680 x1865013828367360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626259.746126:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828367360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626259.746142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626259.746146:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626259.746150:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626259.746156:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367488:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626259.746175:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626259.746183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626259/real 1778626259] req@ffff954342f49f80 x1865013828367488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626259.746191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828367488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626259.746196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626259.746206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626259.746218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367488:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626259.746223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626259.746235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626259.746241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626259/real 1778626259] req@ffff954342f4b100 x1865013828367616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626259.746248:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828367616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626259.746259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626259.746261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626259.746263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626259.746267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367744:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626259.746280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626259.746285:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626259/real 1778626259] req@ffff954342f48a80 x1865013828367744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626259.746293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828367744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626259.746298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626259.746299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626259.746301:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828367744:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626259.937816:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626260.705735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626260.961831:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626261.729761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626261.985851:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626262.753758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626263.009819:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626263.777761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626264.033837:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626264.801762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626264.865905:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626264.865924:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626264.865933:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626264.865953:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10599 00000100:00080000:1.0:1778626264.865973:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626264.865983:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626264.866044:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828367872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626264.866090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626264.866123:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626264.866127:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626264.866137:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10599 00000100:00080000:1.0:1778626264.866144:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626264.866150:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626264.866165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626264.866173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828367872:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626264.866189:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828368000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626264.866206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626264.866211:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626264.866214:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626264.866219:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10599 00000100:00080000:1.0:1778626264.866224:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626264.866229:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626264.866236:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778626264.866245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626264.866255:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828368128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626264.866268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626264.866273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626264.866276:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626264.866283:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10599 00000100:00080000:1.0:1778626264.866288:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626264.866295:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626264/real 1778626264] req@ffff954350197b80 x1865013828367872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626264.866326:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626264.866339:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828367872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778626264.866346:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626264.866366:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828368256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626264.866367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626264.866388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626264.866408:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828367872:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626264.866411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626264.866419:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626264.866424:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626264.866434:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10599 00000100:00100000:2.0:1778626264.866441:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368000:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778626264.866454:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626264.866466:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778626264.866476:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626264.866478:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626264.866488:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626264/real 1778626264] req@ffff954350195880 x1865013828368000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626264.866510:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828368384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626264.866511:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828368000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626264.866607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626264.866619:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626264.866626:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626264.866635:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368128:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626264.866662:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626264.866708:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626264/real 1778626264] req@ffff954350194380 x1865013828368128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626264.866718:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828368128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626264.866723:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626264.866726:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626264.866728:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368128:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626264.866739:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626264.866757:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626264.866764:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626264/real 1778626264] req@ffff954350197800 x1865013828368256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626264.866772:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828368256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626264.866777:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626264.866780:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626264.866782:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626264.866796:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368384:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626264.866806:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626264.866812:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626264/real 1778626264] req@ffff954350195c00 x1865013828368384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626264.866819:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828368384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626264.866824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626264.866826:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626264.866828:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368384:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626265.057881:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626265.825750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626266.081835:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626266.849765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626267.105832:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626267.873731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626268.129838:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626268.897756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626269.153821:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626269.921739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626269.985760:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626269.985768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626269.985772:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626269.985779:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10604 00000100:00080000:3.0:1778626269.985789:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626269.985795:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626269.985826:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828368512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626269.985849:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626269.985851:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626269.985853:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626269.985856:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10604 00000100:00080000:3.0:1778626269.985858:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626269.985860:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626269.985867:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626269.985877:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828368640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626269.985884:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626269.985900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828368512:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626269.985901:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626269.985905:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626269.985910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10604 00000100:00080000:3.0:1778626269.985912:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626269.985916:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626269.985920:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626269.985930:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828368768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626269.985938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626269.985941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626269.985942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626269.985944:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10604 00000100:00080000:3.0:1778626269.985947:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626269.985950:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626269.985952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626269.985959:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626269.985963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828368896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626269.985970:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626269.985973:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626269.985975:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626269.985978:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10604 00000100:00080000:3.0:1778626269.985980:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626269.985982:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626269.985989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626269.985994:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626269/real 1778626269] req@ffff95434701f800 x1865013828368512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626269.986000:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828369024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626269.986017:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828368512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626269.986026:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626269.986030:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626269.986033:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828368512:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626269.986038:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626269.986051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626269.986058:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626269/real 1778626269] req@ffff95434701df80 x1865013828368640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626269.986066:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828368640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626269.986071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626269.986073:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626269.986076:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626269.986093:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368768:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626269.986103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626269.986109:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626269/real 1778626269] req@ffff95434701c000 x1865013828368768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626269.986132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828368768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626269.986140:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626269.986143:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626269.986147:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368768:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626269.986153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626269.986175:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626269.986185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626269/real 1778626269] req@ffff95434701ea00 x1865013828368896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626269.986194:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828368896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626269.986212:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626269.986214:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626269.986217:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828368896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626269.986221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369024:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626269.986236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626269.986242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626269/real 1778626269] req@ffff95434701ca80 x1865013828369024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626269.986250:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828369024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626269.986265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626269.986267:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626269.986269:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369024:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626270.177766:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626270.945750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626271.202446:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626271.969728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626272.225857:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626272.993751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626273.249847:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626274.017724:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626274.273975:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626275.041745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626275.105789:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626275.105799:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626275.105803:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626275.105813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10609 00000100:00080000:3.0:1778626275.105823:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626275.105828:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626275.105861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828369152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626275.105887:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626275.105890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626275.105891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626275.105894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10609 00000100:00080000:3.0:1778626275.105896:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626275.105899:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626275.105906:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626275.105914:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828369280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626275.105920:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626275.105923:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626275.105924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626275.105926:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10609 00000100:00080000:3.0:1778626275.105941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626275.105945:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626275.105949:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626275.105957:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828369408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626275.105971:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828369152:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626275.105975:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626275.105980:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626275.105982:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626275.105985:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10609 00000100:00080000:3.0:1778626275.105987:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626275.105990:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626275.105993:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626275.106002:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828369536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626275.106009:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626275.106011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626275.106012:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626275.106014:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10609 00000100:00080000:3.0:1778626275.106026:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626275.106028:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626275.106031:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626275.106038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828369664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626275.106052:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626275.106087:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626275/real 1778626275] req@ffff95434701fb80 x1865013828369152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626275.106122:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828369152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626275.106131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626275.106134:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626275.106137:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828369152:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626275.106142:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626275.106160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626275.106168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626275/real 1778626275] req@ffff95434701d180 x1865013828369280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626275.106175:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828369280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626275.106190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626275.106192:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626275.106194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626275.106213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369408:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626275.106226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626275.106232:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626275/real 1778626275] req@ffff95434701e680 x1865013828369408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626275.106239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828369408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626275.106244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626275.106246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626275.106248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369408:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626275.106251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626275.106261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626275.106266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626275/real 1778626275] req@ffff95434701f480 x1865013828369536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626275.106273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828369536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626275.106278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626275.106279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626275.106286:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626275.106290:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369664:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626275.106301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626275.106307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626275/real 1778626275] req@ffff95434701ca80 x1865013828369664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626275.106314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828369664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626275.106319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626275.106321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626275.106323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369664:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626275.297853:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626276.065748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626276.321884:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626277.089773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626277.345865:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626278.113779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626278.369883:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626279.137757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626279.393899:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626280.161772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626280.225744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626280.225752:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626280.225756:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626280.225764:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10614 00000100:00080000:3.0:1778626280.225774:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626280.225779:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626280.225809:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828369792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626280.225831:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626280.225833:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626280.225835:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626280.225838:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10614 00000100:00080000:3.0:1778626280.225840:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626280.225842:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626280.225849:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626280.225858:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828369920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626280.225865:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626280.225868:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626280.225870:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626280.225872:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10614 00000100:00080000:3.0:1778626280.225875:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626280.225877:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626280.225886:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828369792:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778626280.225897:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626280.225907:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828370048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626280.225915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626280.225918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626280.225919:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626280.225923:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10614 00000100:00080000:3.0:1778626280.225935:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626280.225939:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778626280.225944:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778626280.225947:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626280.225970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828370176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626280.225983:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626280/real 1778626280] req@ffff95434701ea00 x1865013828369792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626280.225987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626280.226009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626280.226011:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626280.226015:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10614 00000100:00080000:3.0:1778626280.226020:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626280.226024:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828369792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626280.226027:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626280.226034:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626280.226039:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626280.226044:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:3.0:1778626280.226048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828370304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626280.226058:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828369792:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626280.226067:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626280.226088:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626280.226098:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626280/real 1778626280] req@ffff95434701c000 x1865013828369920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626280.226107:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828369920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626280.226113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626280.226116:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626280.226118:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828369920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626280.226137:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370048:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626280.226149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626280.226155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626280/real 1778626280] req@ffff95434701df80 x1865013828370048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626280.226162:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828370048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626280.226167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626280.226170:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626280.226172:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370048:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626280.226175:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626280.226185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626280.226198:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626280/real 1778626280] req@ffff95434701f800 x1865013828370176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626280.226205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828370176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626280.226211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626280.226213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626280.226215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626280.226218:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370304:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626280.226229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626280.226234:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626280/real 1778626280] req@ffff95434701d880 x1865013828370304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626280.226241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828370304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626280.226247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626280.226248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626280.226250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370304:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626280.417788:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626281.185737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626281.441822:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626282.209751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626282.465799:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626283.233719:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626283.489814:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626284.257751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626284.513783:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626285.281727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626285.345743:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626285.345750:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626285.345754:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626285.345762:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10619 00000100:00080000:1.0:1778626285.345771:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626285.345776:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626285.345804:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828370432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626285.345826:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626285.345829:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626285.345830:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626285.345833:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10619 00000100:00080000:1.0:1778626285.345835:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626285.345837:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626285.345845:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626285.345858:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828370560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626285.345863:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828370432:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626285.345879:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626285.345882:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626285.345885:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626285.345912:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10619 00000100:00080000:1.0:1778626285.345918:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626285.345923:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626285.345929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778626285.345936:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626285.345942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828370688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626285.345952:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626285.345954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626285.345956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626285.345960:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10619 00000100:00080000:1.0:1778626285.345963:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626285.345966:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626285.345970:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626285.345978:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626285/real 1778626285] req@ffff954350195500 x1865013828370432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626285.345982:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828370816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626285.345999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626285.346003:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828370432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626285.346006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626285.346011:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626285.346018:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778626285.346020:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10619 00000100:00080000:2.0:1778626285.346025:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778626285.346027:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626285.346032:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828370432:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626285.346036:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626285.346043:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626285.346055:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828370944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626285.346063:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626285.346095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626285.346113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626285/real 1778626285] req@ffff954350194a80 x1865013828370560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626285.346123:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828370560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626285.346129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626285.346132:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626285.346134:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626285.346138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370688:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626285.346151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626285.346158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626285/real 1778626285] req@ffff954350194700 x1865013828370688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626285.346165:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828370688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626285.346180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626285.346183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626285.346185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370688:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626285.346189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626285.346203:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626285.346209:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626285/real 1778626285] req@ffff954350197480 x1865013828370816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626285.346216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828370816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626285.346221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626285.346223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626285.346225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626285.346236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370944:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626285.346246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626285.346251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626285/real 1778626285] req@ffff954350196d80 x1865013828370944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626285.346258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828370944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626285.346263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626285.346265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626285.346271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828370944:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626285.537860:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626286.305794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626286.561858:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626287.329736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626287.585816:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626288.353769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626288.609896:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626289.377756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626289.633825:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626290.401785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626290.465745:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626290.465753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626290.465780:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626290.465788:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10624 00000100:00080000:3.0:1778626290.465812:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626290.465818:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626290.465849:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828371072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626290.465870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626290.465873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626290.465874:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626290.465877:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10624 00000100:00080000:3.0:1778626290.465880:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626290.465882:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626290.465890:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626290.465899:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828371200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626290.465907:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626290.465908:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828371072:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626290.465912:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626290.465915:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626290.465919:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10624 00000100:00080000:3.0:1778626290.465922:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626290.465924:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626290.465928:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626290.465936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828371328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626290.465953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626290.465956:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626290.465957:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626290.465960:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10624 00000100:00080000:3.0:1778626290.465964:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778626290.465969:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626290.465969:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626290.465976:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626290.465987:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828371456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626290.465994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626290.465997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626290.465998:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626290.466001:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10624 00000100:00100000:2.0:1778626290.466005:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626290/real 1778626290] req@ffff95434701d500 x1865013828371072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626290.466005:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626290.466012:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626290.466018:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626290.466028:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828371584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626290.466033:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828371072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626290.466049:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626290.466054:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626290.466059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828371072:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626290.466084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626290.466105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626290.466127:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626290/real 1778626290] req@ffff95434701c700 x1865013828371200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626290.466141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828371200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626290.466150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626290.466153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626290.466166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626290.466173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371328:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626290.466192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626290.466200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626290/real 1778626290] req@ffff95434701f100 x1865013828371328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626290.466208:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828371328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626290.466213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626290.466215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626290.466218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371328:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626290.466221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626290.466232:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626290.466237:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626290/real 1778626290] req@ffff95434701c380 x1865013828371456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626290.466245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828371456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626290.466249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626290.466251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626290.466253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626290.466256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371584:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626290.466265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626290.466277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626290/real 1778626290] req@ffff95434701d880 x1865013828371584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626290.466284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828371584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626290.466290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626290.466292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626290.466294:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371584:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626290.657862:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626291.425787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626291.681837:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626292.449754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626292.705853:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626293.473734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626293.729827:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626294.497776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626294.753907:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626295.521754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626295.585805:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626295.585819:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626295.585825:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626295.585838:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10630 00000100:00080000:0.0:1778626295.585853:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626295.585862:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626295.585911:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828371712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626295.585940:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626295.585961:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626295.585964:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626295.585969:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10630 00000100:00080000:0.0:1778626295.585975:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626295.585981:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828371712:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626295.585982:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626295.585997:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626295.586014:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828371840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626295.586027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626295.586031:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626295.586033:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626295.586038:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10630 00000100:00080000:0.0:1778626295.586042:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626295.586045:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626295.586050:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626295.586061:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828371968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626295.586073:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626295.586073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626295.586080:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626295.586082:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626295.586088:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10630 00000100:00080000:0.0:1778626295.586092:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626295.586097:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626295.586109:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626295.586123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626295/real 1778626295] req@ffff954342f4ad80 x1865013828371712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626295.586125:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828372096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626295.586158:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626295.586162:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626295.586165:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626295.586170:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10630 00000100:00100000:2.0:1778626295.586173:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828371712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626295.586175:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626295.586184:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626295.586192:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626295.586192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626295.586201:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626295.586207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828371712:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626295.586210:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828372224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626295.586245:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626295.586271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626295.586294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626295/real 1778626295] req@ffff954342f49500 x1865013828371840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626295.586308:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828371840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626295.586316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626295.586319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626295.586323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626295.586329:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371968:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626295.586350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626295.586360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626295/real 1778626295] req@ffff954342f49c00 x1865013828371968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626295.586374:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828371968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626295.586382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626295.586385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626295.586389:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828371968:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626295.586395:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626295.586417:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626295.586428:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626295/real 1778626295] req@ffff954342f4b480 x1865013828372096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626295.586440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828372096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626295.586454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626295.586459:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626295.586462:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626295.586469:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372224:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626295.586492:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626295.586503:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626295/real 1778626295] req@ffff954342f4a300 x1865013828372224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626295.586514:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828372224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626295.586523:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626295.586526:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626295.586530:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372224:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626295.777844:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626296.545780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626296.801888:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626297.569760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626297.825874:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626298.593797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626298.849851:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626299.617761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626299.873846:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626300.641791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626300.705777:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626300.705786:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626300.705790:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626300.705799:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10635 00000100:00080000:3.0:1778626300.705827:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626300.705833:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626300.705861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828372352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626300.705885:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626300.705887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626300.705889:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626300.705892:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10635 00000100:00080000:3.0:1778626300.705894:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626300.705896:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626300.705903:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626300.705912:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828372480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626300.705919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626300.705922:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626300.705923:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626300.705925:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10635 00000100:00080000:3.0:1778626300.705927:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626300.705930:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626300.705932:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626300.705942:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828372608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626300.705948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626300.705955:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626300.705956:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626300.705960:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10635 00000100:00080000:3.0:1778626300.705963:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626300.705965:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626300.705969:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626300.705977:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828372736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626300.705986:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626300.705991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626300.705995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626300.706000:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828372352:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626300.706001:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10635 00000100:00080000:3.0:1778626300.706005:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626300.706013:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626300.706024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626300.706034:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828372864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626300.706133:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626300.706199:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626300/real 1778626300] req@ffff95434701f800 x1865013828372352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626300.706230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828372352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626300.706272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626300.706278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626300.706283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828372352:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626300.706292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626300.706323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626300.706338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626300/real 1778626300] req@ffff95434701df80 x1865013828372480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626300.706363:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828372480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626300.706372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626300.706376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626300.706381:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626300.706391:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372608:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626300.706421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626300.706438:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626300/real 1778626300] req@ffff95434701c000 x1865013828372608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626300.706459:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828372608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626300.706473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626300.706479:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626300.706482:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372608:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626300.706539:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626300.706567:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626300.706579:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626300/real 1778626300] req@ffff95434701ea00 x1865013828372736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626300.706592:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828372736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626300.706622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626300.706626:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626300.706629:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626300.706636:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372864:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626300.706659:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626300.706716:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626300/real 1778626300] req@ffff95434701ca80 x1865013828372864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626300.706741:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828372864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626300.706750:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626300.706763:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626300.706766:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828372864:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626300.897873:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626301.665775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626301.921797:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626302.689744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626302.945844:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626303.713788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626303.969805:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626304.737729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626304.993795:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626305.761787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626305.825788:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626305.825796:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626305.825800:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626305.825808:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10640 00000100:00080000:0.0:1778626305.825819:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626305.825827:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626305.825865:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828372992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626305.825898:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626305.825901:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626305.825902:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626305.825905:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10640 00000100:00080000:0.0:1778626305.825926:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626305.825931:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626305.825941:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626305.825953:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828372992:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626305.825959:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828373120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626305.825973:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626305.825976:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626305.825978:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626305.825981:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10640 00000100:00080000:0.0:1778626305.825984:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626305.825987:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626305.825992:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626305.826000:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828373248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626305.826017:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626305.826019:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626305.826021:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626305.826023:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10640 00000100:00080000:0.0:1778626305.826026:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626305.826028:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626305.826031:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626305.826039:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828373376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626305.826055:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626305.826057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626305.826059:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626305.826066:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10640 00000100:00100000:2.0:1778626305.826066:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373120:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778626305.826068:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626305.826082:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626305.826087:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626305.826110:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828373504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626305.826131:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373248:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626305.826155:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626305.826174:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373504:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626306.017915:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626306.785732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626307.041821:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626307.809753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626308.065842:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626308.833721:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778626308.897931:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778626308.897968:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778626308.897994:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626308.898031:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626308.898041:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626308.898051:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626308.898063:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778626308.898078:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626305/real 1778626308] req@ffff954342f49880 x1865013828373120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626308.898113:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828373120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626308.898126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626308.898132:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626308.898135:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626308.898143:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626305/real 1778626308] req@ffff954342f4bb80 x1865013828373248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626308.898157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828373248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626308.898165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626308.898177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626308.898181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373248:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626308.898188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626305/real 1778626308] req@ffff954342f49180 x1865013828373376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626308.898201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828373376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626308.898208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626308.898212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626308.898215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626308.898221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626305/real 1778626308] req@ffff954342f48000 x1865013828372992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626308.898234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828372992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626308.898242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626308.898245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626308.898248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828372992:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626308.898275:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626305/real 1778626308] req@ffff954342f48380 x1865013828373504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626308.898286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828373504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626308.898293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626308.898304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626308.898307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373504:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626309.089848:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626309.857748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626310.113898:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626310.881746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626310.945786:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626310.945797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626310.945800:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626310.945811:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10645 00000100:00080000:1.0:1778626310.945821:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626310.945843:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626310.945876:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828373632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626310.945906:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626310.945911:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626310.945913:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626310.945920:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10645 00000100:00080000:1.0:1778626310.945927:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626310.945934:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626310.945943:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626310.945950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828373632:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626310.945961:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828373760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626310.945988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626310.945993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626310.945995:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626310.946000:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10645 00000100:00080000:1.0:1778626310.946005:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626310.946009:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626310.946015:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778626310.946023:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626310.946031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828373888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626310.946045:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626310.946048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1778626310.946060:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626310/real 1778626310] req@ffff954350194000 x1865013828373632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626310.946062:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626310.946076:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10645 00000100:00080000:1.0:1778626310.946089:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626310.946090:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828373632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626310.946097:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626310.946108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:2.0:1778626310.946111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626310.946118:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626310.946123:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828373632:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626310.946126:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828374016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626310.946140:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626310.946143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626310.946145:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626310.946150:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10645 00000100:00080000:1.0:1778626310.946155:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626310.946155:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373760:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778626310.946169:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626310.946177:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626310.946193:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828374144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626310.946197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626310.946221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626310/real 1778626310] req@ffff954350197100 x1865013828373760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626310.946237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828373760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626310.946245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626310.946249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626310.946253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626310.946267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373888:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626310.946291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626310.946312:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626310/real 1778626310] req@ffff954350196d80 x1865013828373888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626310.946327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828373888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626310.946334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626310.946338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626310.946341:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828373888:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626310.946347:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626310.946368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626310.946380:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626310/real 1778626310] req@ffff954350197480 x1865013828374016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626310.946391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828374016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626310.946414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626310.946418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626310.946421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626310.946440:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374144:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626310.946461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626310.946471:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626310/real 1778626310] req@ffff954350194700 x1865013828374144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626310.946492:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828374144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626310.946500:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626310.946503:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626310.946506:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374144:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626311.137818:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626311.905770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626312.161849:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626312.929755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626313.185804:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626313.953786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626314.209992:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626314.977837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626315.233907:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626316.001782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626316.065792:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626316.065805:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626316.065811:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626316.065825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10650 00000100:00080000:1.0:1778626316.065839:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626316.065847:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626316.065917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828374272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626316.065949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626316.065953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626316.065955:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626316.065959:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10650 00000100:00080000:1.0:1778626316.065963:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626316.065967:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626316.065977:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626316.065988:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828374400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626316.065996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626316.065999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626316.066001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626316.066003:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10650 00000100:00080000:1.0:1778626316.066005:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626316.066008:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626316.066011:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626316.066018:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828374528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626316.066027:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626316.066030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626316.066032:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626316.066042:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10650 00000100:00080000:1.0:1778626316.066046:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626316.066050:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626316.066055:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626316.066061:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828374272:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626316.066069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828374656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626316.066080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626316.066082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626316.066084:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626316.066086:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10650 00000100:00080000:1.0:1778626316.066089:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626316.066092:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626316.066096:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626316.066105:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828374784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626316.066154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626316.066196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626316/real 1778626316] req@ffff954350194a80 x1865013828374272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626316.066227:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828374272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626316.066238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626316.066243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626316.066247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828374272:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626316.066254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626316.066273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626316.066283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626316/real 1778626316] req@ffff954350195500 x1865013828374400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626316.066319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828374400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626316.066327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626316.066330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626316.066333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626316.066338:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374528:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626316.066357:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626316.066365:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626316/real 1778626316] req@ffff954350195c00 x1865013828374528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626316.066376:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828374528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626316.066382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626316.066384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626316.066386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374528:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626316.066415:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626316.066441:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626316.066455:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626316/real 1778626316] req@ffff954350197800 x1865013828374656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626316.066472:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828374656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626316.066499:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626316.066504:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626316.066509:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626316.066517:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374784:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626316.066537:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626316.066547:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626316/real 1778626316] req@ffff954350194380 x1865013828374784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626316.066563:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828374784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626316.066574:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626316.066578:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626316.066583:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828374784:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626316.258012:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626317.025748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626317.281815:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626318.049798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626318.305920:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626319.073751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626319.329776:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626320.097756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626320.353767:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626321.121855:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626321.185764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626321.185772:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626321.185776:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626321.185797:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10655 00000100:00080000:1.0:1778626321.185807:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626321.185813:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626321.185846:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828374912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626321.185867:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626321.185881:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626321.185883:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626321.185886:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10655 00000100:00080000:1.0:1778626321.185889:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626321.185902:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626321.185910:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626321.185919:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828375040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626321.185926:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626321.185928:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626321.185929:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626321.185931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10655 00000100:00080000:1.0:1778626321.185933:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626321.185935:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626321.185937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626321.185944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828375168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626321.185952:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626321.185954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626321.185956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626321.185961:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10655 00000100:00080000:1.0:1778626321.185963:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626321.185965:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626321.185973:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626321.185980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828375296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626321.185989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626321.185991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626321.185992:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626321.185999:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10655 00000100:00080000:1.0:1778626321.186004:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626321.186004:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828374912:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626321.186016:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626321.186026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626321.186046:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828375424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626321.186129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626321.186172:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626321/real 1778626321] req@ffff954350195880 x1865013828374912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626321.186209:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828374912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626321.186233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626321.186237:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626321.186241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828374912:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626321.186248:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626321.186270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626321.186280:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626321/real 1778626321] req@ffff954350197b80 x1865013828375040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626321.186290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828375040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626321.186297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626321.186300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626321.186302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626321.186306:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375168:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626321.186318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626321.186324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626321/real 1778626321] req@ffff954350196a00 x1865013828375168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626321.186333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828375168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626321.186339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626321.186341:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626321.186344:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375168:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626321.186378:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626321.186395:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626321.186403:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626321/real 1778626321] req@ffff954350196680 x1865013828375296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626321.186412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828375296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626321.186418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626321.186421:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626321.186424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626321.186428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375424:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626321.186439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626321.186445:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626321/real 1778626321] req@ffff954350194e00 x1865013828375424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626321.186454:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828375424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626321.186464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626321.186469:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626321.186472:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375424:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626321.377802:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626322.145721:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626322.401778:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626323.169753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626323.425837:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626324.193753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626324.449779:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626325.217735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626325.473830:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626326.241745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626326.305792:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626326.305801:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626326.305804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626326.305813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10660 00000100:00080000:3.0:1778626326.305822:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626326.305828:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626326.305861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828375552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626326.305882:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626326.305885:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626326.305886:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626326.305889:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10660 00000100:00080000:3.0:1778626326.305892:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626326.305894:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626326.305901:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626326.305909:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828375680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626326.305925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626326.305928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626326.305929:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626326.305931:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10660 00000100:00080000:3.0:1778626326.305933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626326.305951:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828375552:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626326.305953:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626326.305960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626326.305970:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828375808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626326.305977:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626326.305979:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626326.305981:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626326.305983:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10660 00000100:00080000:3.0:1778626326.305986:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626326.305998:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626326.306003:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626326.306014:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828375936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626326.306023:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626326.306026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626326.306029:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778626326.306032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626326.306037:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10660 00000100:00080000:3.0:1778626326.306042:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626326.306046:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626326.306051:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626326.306065:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828376064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626326.306084:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626326/real 1778626326] req@ffff95434701f480 x1865013828375552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626326.306125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828375552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626326.306135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626326.306139:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626326.306143:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828375552:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626326.306151:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626326.306173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626326.306185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626326/real 1778626326] req@ffff95434701e680 x1865013828375680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626326.306199:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828375680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626326.306207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626326.306211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626326.306215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626326.306237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375808:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626326.306253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626326.306262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626326/real 1778626326] req@ffff95434701d180 x1865013828375808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626326.306282:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828375808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626326.306291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626326.306294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626326.306298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375808:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626326.306304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626326.306321:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626326.306330:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626326/real 1778626326] req@ffff95434701fb80 x1865013828375936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626326.306342:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828375936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626326.306349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626326.306352:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626326.306355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828375936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626326.306361:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376064:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626326.306378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626326.306385:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626326/real 1778626326] req@ffff95434701ca80 x1865013828376064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626326.306396:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828376064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626326.306414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626326.306418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626326.306421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376064:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626326.497757:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626327.265804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626327.521882:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626328.289738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626328.545780:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626329.313822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626329.569889:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626330.337752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626330.593846:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626331.361770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626331.425776:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626331.425785:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626331.425807:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626331.425817:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10665 00000100:00080000:0.0:1778626331.425829:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626331.425836:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626331.425881:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828376192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626331.425905:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626331.425907:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626331.425908:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626331.425911:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10665 00000100:00080000:0.0:1778626331.425914:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626331.425917:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626331.425924:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626331.425933:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828376320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626331.425940:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626331.425942:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626331.425944:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626331.425946:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10665 00000100:00080000:0.0:1778626331.425949:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626331.425951:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828376192:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626331.425953:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626331.425960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626331.425969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828376448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626331.425985:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626331.425987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626331.426001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626331.426005:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10665 00000100:00080000:0.0:1778626331.426016:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626331.426019:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626331.426023:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626331.426024:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626331.426037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828376576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626331.426054:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626331.426057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626331.426058:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626331.426060:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10665 00000100:00080000:0.0:1778626331.426063:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626331.426066:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626331.426069:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626331.426078:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828376704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626331.426081:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626331/real 1778626331] req@ffff954342f4aa00 x1865013828376192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626331.426111:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828376192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626331.426123:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626331.426128:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626331.426133:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828376192:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626331.426158:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626331.426183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626331.426198:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626331/real 1778626331] req@ffff954342f48e00 x1865013828376320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626331.426213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828376320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626331.426230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626331.426234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626331.426238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626331.426245:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376448:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626331.426269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626331.426281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626331/real 1778626331] req@ffff954342f48380 x1865013828376448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626331.426294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828376448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626331.426302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626331.426306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626331.426310:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376448:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626331.426316:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626331.426339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626331.426351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626331/real 1778626331] req@ffff954342f48000 x1865013828376576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626331.426365:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828376576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626331.426373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626331.426377:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626331.426389:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626331.426397:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376704:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626331.426419:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626331.426431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626331/real 1778626331] req@ffff954342f49180 x1865013828376704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626331.426444:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828376704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626331.426454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626331.426457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626331.426461:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376704:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626331.617842:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626332.385774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626332.641810:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626333.409784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626333.665844:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626334.433733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626334.689777:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626335.457766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626335.713830:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626336.481739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626336.545801:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626336.545808:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626336.545811:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626336.545819:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10671 00000100:00080000:3.0:1778626336.545828:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626336.545833:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626336.545865:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828376832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626336.545901:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626336.545906:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626336.545907:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626336.545911:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10671 00000100:00080000:3.0:1778626336.545914:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626336.545918:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626336.545927:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626336.545939:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828376832:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626336.545939:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828376960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626336.545954:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626336.545956:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626336.545957:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626336.545960:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10671 00000100:00080000:3.0:1778626336.545962:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626336.545964:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626336.545982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626336.545991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828377088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626336.545998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626336.546002:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626336.546003:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626336.546006:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626336.546010:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10671 00000100:00080000:3.0:1778626336.546015:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626336.546019:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626336.546025:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626336.546038:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626336/real 1778626336] req@ffff95434701ea00 x1865013828376832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626336.546040:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828377216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626336.546065:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626336.546069:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828376832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626336.546071:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626336.546075:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626336.546084:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778626336.546085:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10671 00000100:00080000:2.0:1778626336.546090:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778626336.546093:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626336.546098:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626336.546103:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828376832:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778626336.546104:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626336.546117:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828377344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626336.546140:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626336.546167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626336.546181:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626336/real 1778626336] req@ffff95434701c000 x1865013828376960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626336.546193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828376960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626336.546200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626336.546204:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626336.546207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828376960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626336.546213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377088:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626336.546231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626336.546249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626336/real 1778626336] req@ffff95434701df80 x1865013828377088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626336.546261:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828377088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626336.546268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626336.546271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626336.546274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377088:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626336.546280:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626336.546298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626336.546317:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626336/real 1778626336] req@ffff95434701f800 x1865013828377216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626336.546329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828377216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626336.546337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626336.546340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626336.546343:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626336.546349:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377344:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626336.546368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626336.546378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626336/real 1778626336] req@ffff95434701d880 x1865013828377344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626336.546390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828377344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626336.546399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626336.546402:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626336.546405:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377344:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626336.737767:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626337.505733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626337.761992:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626338.529736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626338.785890:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626339.553737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626339.809921:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626340.577740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626340.833947:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626341.601729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626341.665785:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626341.665809:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626341.665814:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626341.665822:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10676 00000100:00080000:3.0:1778626341.665832:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626341.665837:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626341.665880:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828377472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626341.665904:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626341.665907:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626341.665908:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626341.665911:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10676 00000100:00080000:3.0:1778626341.665913:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626341.665915:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626341.665923:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626341.665931:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828377600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626341.665948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626341.665951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626341.665952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626341.665955:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10676 00000100:00080000:3.0:1778626341.665957:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626341.665961:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626341.665963:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828377472:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778626341.665965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626341.665990:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828377728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626341.665998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626341.666000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626341.666001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626341.666004:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10676 00000100:00080000:3.0:1778626341.666019:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626341.666022:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626341.666026:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626341.666032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626341.666037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828377856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626341.666050:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626341.666054:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626341.666056:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626341.666061:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10676 00000100:00080000:3.0:1778626341.666065:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626341.666070:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626341.666075:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626341.666088:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626341/real 1778626341] req@ffff95434701c380 x1865013828377472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626341.666091:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828377984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626341.666124:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828377472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626341.666147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626341.666152:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626341.666156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828377472:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626341.666164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626341.666185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626341.666194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626341/real 1778626341] req@ffff95434701f100 x1865013828377600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626341.666203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828377600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626341.666208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626341.666211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626341.666213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626341.666231:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377728:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626341.666241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626341.666247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626341/real 1778626341] req@ffff95434701c700 x1865013828377728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626341.666254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828377728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626341.666268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626341.666271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626341.666273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377728:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626341.666277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626341.666290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626341.666295:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626341/real 1778626341] req@ffff95434701d500 x1865013828377856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626341.666303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828377856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626341.666318:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626341.666319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626341.666321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626341.666325:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377984:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626341.666333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626341.666338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626341/real 1778626341] req@ffff95434701d880 x1865013828377984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626341.666346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828377984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626341.666351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626341.666352:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626341.666358:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828377984:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626341.857870:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626342.625734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626342.881823:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626343.649763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626343.905853:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626344.673778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626344.929805:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626345.697754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626345.953852:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626346.721784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626346.785742:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626346.785751:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626346.785754:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626346.785762:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10681 00000100:00080000:2.0:1778626346.785771:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626346.785776:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626346.785806:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828378112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626346.785829:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626346.785832:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626346.785833:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626346.785836:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10681 00000100:00080000:2.0:1778626346.785839:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626346.785841:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626346.785848:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626346.785857:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828378240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626346.785864:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626346.785877:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828378112:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626346.785878:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626346.785881:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626346.785885:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10681 00000100:00080000:2.0:1778626346.785888:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626346.785894:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626346.785898:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626346.785915:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828378368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626346.785924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626346.785926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626346.785927:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626346.785930:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10681 00000100:00080000:2.0:1778626346.785933:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626346.785936:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626346.785939:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626346.785945:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626346.785949:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828378496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626346.785957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626346.785959:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626346.785961:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626346.785963:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10681 00000100:00080000:2.0:1778626346.785966:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626346.785969:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626346.785977:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626346/real 1778626346] req@ffff95435878aa00 x1865013828378112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778626346.785980:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626346.785996:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828378624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626346.785998:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828378112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626346.786027:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626346.786032:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626346.786036:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828378112:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626346.786042:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626346.786056:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626346.786063:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626346/real 1778626346] req@ffff95435878b480 x1865013828378240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626346.786071:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828378240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626346.786076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626346.786078:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626346.786080:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626346.786100:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378368:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626346.786118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626346.786129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626346/real 1778626346] req@ffff95435878b100 x1865013828378368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626346.786156:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828378368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626346.786164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626346.786166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626346.786168:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378368:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626346.786172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626346.786188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626346.786195:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626346/real 1778626346] req@ffff954358788380 x1865013828378496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626346.786203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828378496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626346.786208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626346.786210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626346.786211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626346.786215:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378624:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626346.786223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626346.786228:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626346/real 1778626346] req@ffff95435878bb80 x1865013828378624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626346.786235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828378624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626346.786248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626346.786250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626346.786252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378624:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626346.977823:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626347.745748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626348.001826:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626348.769871:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626349.025831:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626349.793730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626350.049809:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626350.817748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626351.073812:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626351.841734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626351.905753:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626351.905761:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626351.905765:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626351.905773:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10686 00000100:00080000:1.0:1778626351.905792:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626351.905798:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626351.905832:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828378752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626351.905861:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626351.905866:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626351.905868:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626351.905872:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10686 00000100:00080000:1.0:1778626351.905876:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626351.905880:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626351.905891:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828378752:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778626351.905894:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626351.905909:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828378880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626351.905917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626351.905920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626351.905921:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626351.905924:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10686 00000100:00080000:1.0:1778626351.905927:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626351.905930:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626351.905933:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626351.905941:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828379008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626351.905947:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626351.905957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1778626351.905958:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626351.905959:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626351.905970:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10686 00000100:00080000:1.0:1778626351.905975:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626351.905978:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626351.905982:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626351.905992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828379136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626351.905999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626351.906000:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626351/real 1778626351] req@ffff954350195180 x1865013828378752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626351.906002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626351.906004:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626351.906008:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10686 00000100:00080000:1.0:1778626351.906013:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626351.906020:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626351.906026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626351.906034:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828378752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626351.906040:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828379264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626351.906065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626351.906071:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626351.906076:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828378752:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626351.906101:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626351.906127:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626351.906142:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626351/real 1778626351] req@ffff954350195f80 x1865013828378880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626351.906154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828378880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626351.906179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626351.906183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626351.906187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828378880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626351.906194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379008:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626351.906217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626351.906239:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626351/real 1778626351] req@ffff954350194e00 x1865013828379008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626351.906250:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828379008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626351.906258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626351.906261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626351.906264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379008:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626351.906268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626351.906291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626351.906302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626351/real 1778626351] req@ffff954350196680 x1865013828379136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626351.906313:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828379136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626351.906321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626351.906325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626351.906333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626351.906338:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379264:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626351.906359:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626351.906368:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626351/real 1778626351] req@ffff954350196a00 x1865013828379264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626351.906376:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828379264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626351.906381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626351.906383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626351.906385:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379264:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626352.097808:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626352.865758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626353.121822:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626353.889739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626354.145816:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626354.913749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626355.169819:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626355.937750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626356.193829:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626356.961759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626357.025750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626357.025757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626357.025761:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626357.025769:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10691 00000100:00080000:1.0:1778626357.025790:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626357.025795:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626357.025823:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828379392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626357.025847:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626357.025850:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626357.025852:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626357.025855:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10691 00000100:00080000:1.0:1778626357.025857:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626357.025859:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626357.025866:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626357.025878:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828379520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626357.025890:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626357.025893:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828379392:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626357.025895:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626357.025899:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626357.025905:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10691 00000100:00080000:1.0:1778626357.025910:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626357.025915:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626357.025920:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626357.025935:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828379648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626357.025943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626357.025952:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626357.025953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626357.025956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626357.025960:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10691 00000100:00080000:1.0:1778626357.025965:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626357.025968:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626357.025973:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626357.025982:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828379776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626357.025990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626357.025992:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626357/real 1778626357] req@ffff954350197b80 x1865013828379392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626357.025993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626357.025995:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626357.026001:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10691 00000100:00080000:1.0:1778626357.026006:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626357.026011:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626357.026016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626357.026023:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828379904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626357.026029:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828379392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626357.026047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626357.026052:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626357.026064:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828379392:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626357.026084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626357.026102:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626357.026111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626357/real 1778626357] req@ffff954350195880 x1865013828379520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626357.026121:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828379520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626357.026126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626357.026129:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626357.026131:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626357.026135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379648:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626357.026145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626357.026150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626357/real 1778626357] req@ffff954350194380 x1865013828379648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626357.026158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828379648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626357.026163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626357.026164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626357.026166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379648:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626357.026170:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626357.026180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626357.026192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626357/real 1778626357] req@ffff954350197800 x1865013828379776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626357.026200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828379776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626357.026204:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626357.026206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626357.026208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626357.026212:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379904:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626357.026222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626357.026227:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626357/real 1778626357] req@ffff954350195c00 x1865013828379904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626357.026234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828379904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626357.026239:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626357.026241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626357.026242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828379904:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626357.217816:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626357.985760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626358.241850:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626359.009785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626359.265821:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626360.033793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626360.289817:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626361.057732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626361.313811:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626362.081764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626362.145755:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626362.145762:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626362.145766:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626362.145774:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10696 00000100:00080000:3.0:1778626362.145783:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626362.145789:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626362.145820:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828380032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626362.145843:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626362.145846:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626362.145847:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626362.145850:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10696 00000100:00080000:3.0:1778626362.145852:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626362.145855:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626362.145862:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626362.145871:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828380160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626362.145879:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626362.145883:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778626362.145885:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828380032:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626362.145886:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626362.145903:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10696 00000100:00080000:3.0:1778626362.145907:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626362.145910:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626362.145914:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626362.145922:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828380288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626362.145929:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626362.145931:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626362.145932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626362.145935:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10696 00000100:00080000:3.0:1778626362.145938:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626362.145940:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626362.145943:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626362.145947:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626362.145952:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828380416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626362.145961:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626362.145965:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626362.145967:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626362.145972:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10696 00000100:00080000:3.0:1778626362.145976:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626362.145980:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626362.145985:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626362.145990:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626362/real 1778626362] req@ffff95434701f800 x1865013828380032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626362.146001:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828380544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626362.146008:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828380032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626362.146040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626362.146044:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626362.146048:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828380032:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626362.146067:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626362.146083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626362.146091:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626362/real 1778626362] req@ffff95434701df80 x1865013828380160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626362.146100:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828380160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626362.146105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626362.146107:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626362.146110:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626362.146113:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380288:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626362.146123:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626362.146128:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626362/real 1778626362] req@ffff95434701c000 x1865013828380288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626362.146143:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828380288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626362.146157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626362.146160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626362.146162:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380288:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626362.146166:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626362.146181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626362.146187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626362/real 1778626362] req@ffff95434701ea00 x1865013828380416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626362.146195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828380416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626362.146200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626362.146202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626362.146204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626362.146207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380544:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626362.146216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626362.146221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626362/real 1778626362] req@ffff95434701ca80 x1865013828380544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626362.146228:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828380544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626362.146233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626362.146235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626362.146241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380544:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626362.337811:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626363.105730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626363.361849:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626364.129753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626364.385847:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626365.153790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626365.409816:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626366.177776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626366.433923:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626367.201734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626367.265819:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626367.265832:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626367.265863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626367.265877:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10701 00000100:00080000:1.0:1778626367.265892:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626367.265900:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626367.265946:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828380672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626367.265982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626367.265986:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626367.265988:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626367.265993:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10701 00000100:00080000:1.0:1778626367.265999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626367.266004:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626367.266017:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626367.266033:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828380800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626367.266040:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828380672:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626367.266048:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626367.266057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626367.266058:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626367.266064:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10701 00000100:00080000:1.0:1778626367.266068:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626367.266073:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626367.266079:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626367.266092:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828380928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626367.266116:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626367.266120:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626367.266122:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778626367.266130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626367.266131:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10701 00000100:00080000:1.0:1778626367.266142:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626367.266148:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626367.266156:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626367.266174:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828381056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626367.266183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626367/real 1778626367] req@ffff954350195500 x1865013828380672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626367.266196:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626367.266211:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626367.266214:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626367.266221:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10701 00000100:00100000:2.0:1778626367.266222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828380672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626367.266230:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626367.266241:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778626367.266246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778626367.266249:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626367.266252:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626367.266261:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828380672:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626367.266271:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828381184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626367.266366:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626367.266387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626367.266398:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626367/real 1778626367] req@ffff954350194a80 x1865013828380800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626367.266407:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828380800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626367.266414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626367.266417:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626367.266436:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626367.266443:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380928:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626367.266466:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626367.266478:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626367/real 1778626367] req@ffff954350194700 x1865013828380928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626367.266492:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828380928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626367.266500:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626367.266504:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626367.266508:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828380928:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626367.266515:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626367.266535:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626367.266543:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626367/real 1778626367] req@ffff954350197480 x1865013828381056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626367.266552:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828381056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626367.266558:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626367.266560:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626367.266562:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626367.266566:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381184:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626367.266576:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626367.266590:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626367/real 1778626367] req@ffff954350196d80 x1865013828381184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626367.266598:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828381184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626367.266604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626367.266606:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626367.266608:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381184:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626367.457809:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626368.225759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626368.481830:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626369.249737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626369.505792:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626370.273745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626370.529813:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626371.297739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626371.553816:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626372.321778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626372.385775:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626372.385785:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626372.385790:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626372.385811:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10706 00000100:00080000:2.0:1778626372.385822:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626372.385830:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626372.385882:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828381312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626372.385906:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626372.385909:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626372.385911:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626372.385914:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10706 00000100:00080000:2.0:1778626372.385917:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626372.385935:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626372.385942:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626372.385962:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828381440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626372.385969:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626372.385972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626372.385973:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626372.385976:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10706 00000100:00080000:2.0:1778626372.385980:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626372.385982:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626372.385986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626372.385992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828381568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626372.386000:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626372.386002:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626372.386004:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626372.386007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10706 00000100:00080000:2.0:1778626372.386009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626372.386012:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626372.386014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626372.386024:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828381312:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626372.386024:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828381696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626372.386038:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626372.386045:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626372.386046:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626372.386049:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10706 00000100:00080000:2.0:1778626372.386052:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626372.386055:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626372.386058:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626372.386069:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828381824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626372.386226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626372.386264:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381568:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626372.386293:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626372.386309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381824:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626372.577934:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626373.345736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626373.601822:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626374.369757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626374.625836:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626375.393737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778626375.457890:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778626375.457944:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778626375.457961:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626375.458003:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626375.458013:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626375.458021:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626375.458028:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626375.458314:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626372/real 1778626375] req@ffff954358788700 x1865013828381696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626375.458342:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828381696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626375.458354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626375.458359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626375.458372:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626375.458380:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626372/real 1778626375] req@ffff954358789880 x1865013828381824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626375.458410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828381824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626375.458420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626375.458424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626375.458428:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381824:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626375.458435:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626372/real 1778626375] req@ffff954358788a80 x1865013828381312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626375.458444:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828381312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626375.458449:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626375.458452:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626375.458454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828381312:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626375.458460:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626372/real 1778626375] req@ffff954358789500 x1865013828381440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626375.458471:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828381440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626375.458478:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626375.458482:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626375.458494:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626375.458503:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626372/real 1778626375] req@ffff954358788e00 x1865013828381568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626375.458517:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828381568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626375.458524:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626375.458527:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626375.458530:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828381568:192.168.204.102@tcp:38:kworker.0 00010000:00010000:0.0:1778626375.649847:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626376.417724:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626376.673832:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626377.441768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626377.505814:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626377.505822:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626377.505826:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626377.505834:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10712 00000100:00080000:1.0:1778626377.505844:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626377.505849:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626377.505882:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828381952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626377.505926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626377.505930:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626377.505932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626377.505937:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10712 00000100:00080000:1.0:1778626377.505941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626377.505945:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626377.505956:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626377.505973:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828382080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626377.505985:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828381952:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626377.505987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626377.506006:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626377.506008:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626377.506013:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10712 00000100:00080000:1.0:1778626377.506024:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626377.506029:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626377.506035:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626377.506054:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828382208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626377.506055:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626377.506069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626377.506073:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626377.506076:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626377.506093:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10712 00000100:00080000:1.0:1778626377.506099:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626377.506104:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626377.506110:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626377/real 1778626377] req@ffff954350197100 x1865013828381952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778626377.506111:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626377.506130:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828382336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626377.506137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828381952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626377.506147:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626377.506157:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626377.506159:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626377.506161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778626377.506166:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10712 00000100:00080000:0.0:1778626377.506171:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778626377.506173:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626377.506179:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828381952:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626377.506181:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626377.506195:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626377.506208:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778626377.506213:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828382464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626377.506244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626377.506260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626377/real 1778626377] req@ffff954350194000 x1865013828382080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626377.506290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828382080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626377.506300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626377.506304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626377.506307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626377.506314:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382208:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626377.506337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626377.506349:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626377/real 1778626377] req@ffff954350196d80 x1865013828382208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626377.506362:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828382208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626377.506371:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626377.506374:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626377.506378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382208:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626377.506383:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626377.506402:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626377.506413:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626377/real 1778626377] req@ffff954350197480 x1865013828382336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626377.506425:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828382336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626377.506447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626377.506451:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626377.506454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626377.506474:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382464:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626377.506497:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626377.506509:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626377/real 1778626377] req@ffff954350194700 x1865013828382464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626377.506521:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828382464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626377.506530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626377.506533:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626377.506536:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382464:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626377.697812:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626378.465739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626378.721834:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626379.489747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626379.745776:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626380.513727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626380.769832:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626381.537776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626381.793796:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626382.561767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626382.625771:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626382.625779:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626382.625783:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626382.625792:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10717 00000100:00080000:2.0:1778626382.625801:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626382.625806:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626382.625839:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788000 x1865013828382592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626382.625881:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626382.625885:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626382.625887:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626382.625892:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10717 00000100:00080000:2.0:1778626382.625897:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626382.625901:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626382.625911:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626382.625929:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828382592:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626382.625932:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828382720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626382.625942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626382.625945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626382.625946:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626382.625950:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10717 00000100:00080000:2.0:1778626382.625953:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626382.625956:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626382.625961:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626382.625969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828382848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626382.625977:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626382.625996:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626382.625998:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778626382.626002:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626382.626002:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10717 00000100:00080000:2.0:1778626382.626012:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626382.626016:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626382.626022:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626382.626032:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828382976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626382.626037:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626382/real 1778626382] req@ffff954358788000 x1865013828382592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626382.626042:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626382.626046:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626382.626048:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626382.626051:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10717 00000100:00100000:0.0:1778626382.626054:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828382592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626382.626054:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626382.626057:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626382.626061:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626382.626070:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828383104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626382.626085:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626382.626090:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626382.626094:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828382592:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626382.626118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626382.626137:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626382.626147:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626382/real 1778626382] req@ffff954358789180 x1865013828382720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626382.626155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828382720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626382.626160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626382.626162:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626382.626164:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626382.626169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382848:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626382.626179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626382.626184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626382/real 1778626382] req@ffff954358789c00 x1865013828382848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626382.626192:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828382848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626382.626197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626382.626198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626382.626200:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382848:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626382.626209:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626382.626222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626382.626228:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626382/real 1778626382] req@ffff95435878b800 x1865013828382976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626382.626235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828382976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626382.626241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626382.626243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626382.626245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828382976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626382.626251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383104:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626382.626268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626382.626278:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626382/real 1778626382] req@ffff95435878a300 x1865013828383104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626382.626290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828383104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626382.626299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626382.626312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626382.626315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383104:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626382.817856:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626383.585782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626383.841778:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626384.609740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626384.865867:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626385.633753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626385.889778:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626386.657750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626386.913792:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626387.681761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626387.745762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626387.745771:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626387.745775:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626387.745782:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10722 00000100:00080000:3.0:1778626387.745792:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626387.745797:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626387.745829:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828383232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626387.745853:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626387.745856:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626387.745857:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626387.745860:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10722 00000100:00080000:3.0:1778626387.745863:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626387.745865:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626387.745872:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626387.745893:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828383360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626387.745901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626387.745904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626387.745905:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626387.745907:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10722 00000100:00080000:3.0:1778626387.745910:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626387.745912:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626387.745931:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626387.745940:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828383488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626387.745947:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626387.745950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828383232:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626387.745950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626387.745953:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626387.745959:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10722 00000100:00080000:3.0:1778626387.745961:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626387.745965:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626387.745973:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626387.745983:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828383616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626387.745992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626387.745996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626387.745998:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626387.746002:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10722 00000100:00080000:3.0:1778626387.746007:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626387.746011:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626387.746016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626387.746023:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626387.746030:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828383744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626387.746058:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626387/real 1778626387] req@ffff95434701fb80 x1865013828383232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626387.746074:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828383232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626387.746081:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626387.746094:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626387.746097:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828383232:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626387.746102:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626387.746116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626387.746124:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626387/real 1778626387] req@ffff95434701d180 x1865013828383360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626387.746132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828383360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626387.746137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626387.746139:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626387.746141:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626387.746158:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383488:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626387.746167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626387.746173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626387/real 1778626387] req@ffff95434701e680 x1865013828383488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626387.746190:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828383488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626387.746196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626387.746198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626387.746200:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383488:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626387.746203:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626387.746215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626387.746221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626387/real 1778626387] req@ffff95434701f480 x1865013828383616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626387.746229:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828383616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626387.746233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626387.746235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626387.746237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626387.746240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383744:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626387.746249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626387.746254:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626387/real 1778626387] req@ffff95434701ca80 x1865013828383744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626387.746261:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828383744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626387.746274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626387.746276:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626387.746278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828383744:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626387.937801:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626388.705779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626388.961928:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626389.729755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626389.985791:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626390.753733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626391.009839:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626391.777760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626392.033851:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626392.801761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626392.865915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626392.865925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626392.865945:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626392.865956:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10727 00000100:00080000:3.0:1778626392.865967:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626392.865974:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626392.866004:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828383872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626392.866089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828383872:192.168.204.102@tcp:250:kworker.0 00000800:00000100:0.0:1778626392.866161:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626392.866194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626392.866206:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626392.866208:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626392.866214:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10727 00000100:00080000:3.0:1778626392.866218:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626392.866223:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626392.866232:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626392.866245:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828384000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626392.866303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626392/real 1778626392] req@ffff95434701ea00 x1865013828383872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626392.866321:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828383872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626392.866331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626392.866335:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626392.866340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828383872:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626392.866372:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626392.866398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626392.866411:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626392/real 1778626392] req@ffff95434701c000 x1865013828384000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626392.866425:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828384000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626392.866449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626392.866453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778626392.866454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626392.866456:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626392.866457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778626392.866460:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10727 00000100:00100000:0.0:1778626392.866461:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384000:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778626392.866465:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626392.866469:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626392.866474:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626392.866488:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828384128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626392.866517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626392.866520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626392.866522:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626392.866535:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10727 00000100:00080000:3.0:1778626392.866543:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626392.866544:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384128:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778626392.866548:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626392.866553:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626392.866566:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828384256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626392.866567:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626392.866576:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626392.866580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626392.866581:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626392.866585:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10727 00000100:00100000:0.0:1778626392.866587:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626392/real 1778626392] req@ffff95434701df80 x1865013828384128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626392.866588:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626392.866592:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626392.866596:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626392.866599:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828384128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626392.866607:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828384384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626392.866608:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626392.866612:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626392.866615:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384128:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626392.866627:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626392.866648:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626392.866660:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626392/real 1778626392] req@ffff95434701f800 x1865013828384256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626392.866668:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828384256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626392.866689:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626392.866691:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626392.866699:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626392.866703:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384384:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626392.866719:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626392.866726:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626392/real 1778626392] req@ffff95434701d880 x1865013828384384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626392.866734:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828384384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626392.866739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626392.866741:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626392.866743:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384384:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626393.057889:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626393.825778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626394.081902:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626394.849804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626395.105869:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626395.873744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626396.129802:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626396.897759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626397.153817:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626397.921742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626397.985744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626397.985752:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626397.985756:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626397.985765:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10732 00000100:00080000:3.0:1778626397.985792:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626397.985797:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626397.985826:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828384512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626397.985850:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626397.985853:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626397.985854:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626397.985857:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10732 00000100:00080000:3.0:1778626397.985860:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626397.985862:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626397.985870:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626397.985878:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828384640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626397.985886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626397.985888:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626397.985889:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626397.985892:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10732 00000100:00080000:3.0:1778626397.985894:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626397.985896:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626397.985900:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626397.985901:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828384512:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626397.985908:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828384768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626397.985915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626397.985922:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626397.985924:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626397.985927:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10732 00000100:00080000:3.0:1778626397.985930:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626397.985933:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626397.985937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626397.985945:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828384896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626397.985951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626397.985954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626397.985955:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626397.985958:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10732 00000100:00080000:3.0:1778626397.985960:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626397.985961:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626397.985965:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626397.985969:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626397.985976:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828385024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626397.986004:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626397/real 1778626397] req@ffff95434701d500 x1865013828384512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626397.986018:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828384512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626397.986028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626397.986032:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626397.986036:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828384512:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626397.986045:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626397.986078:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626397.986090:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626397/real 1778626397] req@ffff95434701c700 x1865013828384640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626397.986099:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828384640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626397.986106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626397.986108:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626397.986110:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626397.986127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384768:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626397.986140:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626397.986146:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626397/real 1778626397] req@ffff95434701f100 x1865013828384768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626397.986154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828384768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626397.986159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626397.986162:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626397.986163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384768:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626397.986167:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626397.986176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626397.986181:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626397/real 1778626397] req@ffff95434701c380 x1865013828384896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626397.986204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828384896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626397.986212:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626397.986215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626397.986219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828384896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626397.986225:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385024:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626397.986246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626397.986256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626397/real 1778626397] req@ffff95434701d880 x1865013828385024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626397.986264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828385024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626397.986269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626397.986271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626397.986273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385024:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626398.177812:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626398.945749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626399.201810:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626399.969751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626400.225853:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626400.993788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626401.249786:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626402.017748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626402.273818:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626403.041762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626403.105744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626403.105754:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626403.105759:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626403.105770:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10737 00000100:00080000:3.0:1778626403.105783:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626403.105789:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626403.105832:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828385152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626403.105882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626403.105885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626403.105887:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626403.105890:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10737 00000100:00080000:3.0:1778626403.105893:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626403.105896:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626403.105907:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626403.105920:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828385280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626403.105932:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626403.105937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626403.105939:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626403.105944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10737 00000100:00080000:3.0:1778626403.105949:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626403.105953:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626403.105959:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626403.105972:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828385408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626403.105976:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828385152:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626403.105982:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626403.105991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626403.105993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626403.105997:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10737 00000100:00080000:3.0:1778626403.106005:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626403.106010:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626403.106015:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626403.106024:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828385536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626403.106032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626403.106036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626403.106038:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626403.106040:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10737 00000100:00080000:3.0:1778626403.106042:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626403.106044:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626403.106048:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626403.106055:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828385664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626403.106065:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626403.106112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626403/real 1778626403] req@ffff95434701f800 x1865013828385152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626403.106154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828385152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626403.106163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626403.106166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626403.106169:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828385152:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626403.106175:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626403.106195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626403.106203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626403/real 1778626403] req@ffff95434701df80 x1865013828385280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626403.106210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828385280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626403.106216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626403.106218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626403.106220:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626403.106239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385408:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626403.106249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626403.106255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626403/real 1778626403] req@ffff95434701c000 x1865013828385408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626403.106262:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828385408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626403.106275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626403.106278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626403.106280:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385408:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626403.106284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626403.106298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626403.106303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626403/real 1778626403] req@ffff95434701ea00 x1865013828385536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626403.106311:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828385536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626403.106315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626403.106317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626403.106319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626403.106323:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385664:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626403.106332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626403.106337:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626403/real 1778626403] req@ffff95434701ca80 x1865013828385664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626403.106344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828385664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626403.106349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626403.106351:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626403.106357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385664:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626403.297850:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626404.065733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626404.322153:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626405.089739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626405.346528:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626406.113751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626406.370659:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626407.137754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626407.393876:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626408.161831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626408.225794:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626408.225806:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626408.225812:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626408.225825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10742 00000100:00080000:0.0:1778626408.225840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626408.225849:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626408.225901:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828385792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626408.225935:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626408.225938:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626408.225940:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626408.225944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10742 00000100:00080000:0.0:1778626408.225947:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626408.225951:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626408.225961:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626408.225973:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828385920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626408.225983:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626408.225983:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828385792:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626408.225989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626408.225992:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626408.226014:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10742 00000100:00080000:0.0:1778626408.226018:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626408.226021:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626408.226026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626408.226035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828386048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626408.226043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626408.226046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626408.226048:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626408.226050:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10742 00000100:00080000:0.0:1778626408.226053:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626408.226055:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626408.226058:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626408.226065:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828386176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626408.226072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626408.226074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626408.226075:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626408.226080:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10742 00000800:00000100:2.0:1778626408.226083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626408.226084:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626408.226088:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626408.226091:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626408.226098:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828386304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626408.226135:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626408/real 1778626408] req@ffff954342f4bb80 x1865013828385792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626408.226160:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828385792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626408.226171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626408.226176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626408.226180:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828385792:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626408.226210:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626408.226234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626408.226246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626408/real 1778626408] req@ffff954342f49880 x1865013828385920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626408.226259:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828385920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626408.226277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626408.226282:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626408.226285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828385920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626408.226291:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386048:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626408.226316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626408.226327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626408/real 1778626408] req@ffff954342f4a300 x1865013828386048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626408.226340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828386048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626408.226348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626408.226352:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626408.226355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386048:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626408.226362:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626408.226382:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626408.226390:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626408/real 1778626408] req@ffff954342f4b480 x1865013828386176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626408.226399:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828386176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626408.226404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626408.226406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626408.226414:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626408.226419:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386304:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626408.226434:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626408.226440:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626408/real 1778626408] req@ffff954342f49c00 x1865013828386304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626408.226447:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828386304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626408.226453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626408.226456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626408.226459:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386304:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626408.417884:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626409.185784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626409.441912:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626410.209756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626410.465827:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626411.233786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626411.489798:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626412.257772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626412.513807:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626413.281759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626413.345763:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626413.345770:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626413.345774:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626413.345782:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10747 00000100:00080000:0.0:1778626413.345792:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626413.345798:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626413.345834:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828386432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626413.345868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626413.345870:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626413.345872:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626413.345875:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10747 00000100:00080000:0.0:1778626413.345877:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626413.345879:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626413.345885:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626413.345895:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828386560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626413.345907:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626413.345910:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626413.345924:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626413.345927:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10747 00000100:00080000:0.0:1778626413.345930:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626413.345939:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626413.345954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626413.345957:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828386432:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626413.345967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828386688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626413.345976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626413.345978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626413.345979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626413.345982:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10747 00000100:00080000:0.0:1778626413.345984:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626413.345987:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626413.345990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626413.345997:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828386816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626413.346003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626413.346006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626413.346007:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626413.346011:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10747 00000100:00080000:0.0:1778626413.346013:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626413.346015:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626413.346018:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626413.346030:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828386944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626413.346041:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626413.346086:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626413/real 1778626413] req@ffff954342f49500 x1865013828386432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626413.346103:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828386432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626413.346113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626413.346117:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626413.346121:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828386432:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626413.346132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626413.346159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626413.346191:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626413/real 1778626413] req@ffff954342f4ad80 x1865013828386560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626413.346218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828386560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626413.346231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626413.346234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626413.346237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626413.346263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386688:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626413.346280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626413.346288:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626413/real 1778626413] req@ffff954342f48a80 x1865013828386688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626413.346297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828386688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626413.346303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626413.346305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626413.346308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386688:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626413.346328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626413.346341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626413.346348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626413/real 1778626413] req@ffff954342f4b100 x1865013828386816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626413.346355:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828386816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626413.346367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626413.346370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626413.346372:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626413.346393:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386944:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626413.346411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626413.346435:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626413/real 1778626413] req@ffff954342f49f80 x1865013828386944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626413.346445:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828386944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626413.346451:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626413.346454:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626413.346456:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828386944:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626413.537811:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626414.305744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626414.561802:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626415.329801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626415.585865:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626416.353739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626416.609845:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626417.377754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626417.633852:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626418.401773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626418.465811:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626418.465822:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626418.465827:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626418.465837:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10752 00000100:00080000:2.0:1778626418.465851:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626418.465857:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626418.465899:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828387072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626418.465927:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626418.465945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626418.465947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626418.465951:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10752 00000100:00080000:2.0:1778626418.465954:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626418.465958:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626418.465972:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626418.465985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828387200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626418.465987:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828387072:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626418.466016:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626418.466020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626418.466023:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626418.466029:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10752 00000100:00080000:2.0:1778626418.466034:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626418.466038:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626418.466044:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626418.466057:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828387328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626418.466065:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626418.466073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626418.466074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626418.466078:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626418.466084:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10752 00000100:00080000:2.0:1778626418.466091:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626418.466097:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626418.466111:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626418.466117:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626418/real 1778626418] req@ffff95435878ad80 x1865013828387072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626418.466127:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828387456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626418.466149:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626418.466155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828387072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626418.466155:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626418.466162:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626418.466175:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10752 00000100:00080000:2.0:1778626418.466186:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626418.466192:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626418.466193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626418.466204:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:2.0:1778626418.466204:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626418.466213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828387072:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626418.466219:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828387584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626418.466242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626418.466269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626418.466294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626418/real 1778626418] req@ffff95435878a680 x1865013828387200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626418.466303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828387200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626418.466310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626418.466323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626418.466326:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626418.466339:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387328:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626418.466355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626418.466362:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626418/real 1778626418] req@ffff954358789f80 x1865013828387328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626418.466370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828387328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626418.466375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626418.466377:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626418.466379:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387328:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626418.466383:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626418.466393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626418.466399:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626418/real 1778626418] req@ffff95435878a300 x1865013828387456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626418.466406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828387456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626418.466411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626418.466412:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626418.466414:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626418.466418:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387584:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626418.466435:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626418.466442:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626418/real 1778626418] req@ffff95435878b800 x1865013828387584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626418.466454:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828387584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626418.466463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626418.466467:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626418.466470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387584:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626418.657805:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626419.425755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626419.681817:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626420.449763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626420.705815:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626421.473755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626421.729831:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626422.497742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626422.753793:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626423.521748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626423.585785:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626423.585793:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626423.585797:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626423.585805:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10758 00000100:00080000:1.0:1778626423.585815:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626423.585834:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626423.585861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828387712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626423.585882:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626423.585885:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626423.585887:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626423.585890:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10758 00000100:00080000:1.0:1778626423.585893:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626423.585895:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626423.585903:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626423.585911:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828387840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626423.585919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626423.585924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778626423.585925:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828387712:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626423.585925:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626423.585933:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10758 00000100:00080000:1.0:1778626423.585935:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626423.585937:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626423.585941:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626423.585950:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828387968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626423.585957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626423.585959:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626423.585965:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626423.585969:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10758 00000100:00080000:1.0:1778626423.585975:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626423.585978:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626423.585982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626423.585986:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626423.585995:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828388096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626423.586003:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626423.586006:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626423.586007:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626423.586010:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10758 00000100:00080000:1.0:1778626423.586013:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626423.586017:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626423.586019:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626423/real 1778626423] req@ffff954350194a80 x1865013828387712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778626423.586021:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626423.586038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828388224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626423.586048:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828387712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626423.586076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626423.586080:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626423.586085:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828387712:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626423.586103:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626423.586128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626423.586138:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626423/real 1778626423] req@ffff954350195500 x1865013828387840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626423.586146:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828387840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626423.586152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626423.586154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626423.586156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626423.586160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387968:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626423.586169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626423.586189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626423/real 1778626423] req@ffff954350195c00 x1865013828387968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626423.586197:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828387968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626423.586202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626423.586205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626423.586207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828387968:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626423.586210:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626423.586224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626423.586230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626423/real 1778626423] req@ffff954350197800 x1865013828388096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626423.586247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828388096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626423.586253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626423.586255:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626423.586256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626423.586260:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388224:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626423.586272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626423.586277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626423/real 1778626423] req@ffff954350194380 x1865013828388224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626423.586284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828388224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626423.586290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626423.586292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626423.586293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388224:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626423.777802:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626424.545809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626424.801825:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626425.569769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626425.825834:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626426.593756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626426.849814:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626427.617736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626427.873798:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626428.641732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626428.705745:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626428.705752:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626428.705756:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626428.705765:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10763 00000100:00080000:1.0:1778626428.705774:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626428.705779:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626428.705826:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828388352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626428.705851:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626428.705853:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626428.705855:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626428.705857:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10763 00000100:00080000:1.0:1778626428.705860:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626428.705863:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626428.705870:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626428.705879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828388480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626428.705888:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626428.705890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626428.705892:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626428.705895:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828388352:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626428.705897:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10763 00000100:00080000:1.0:1778626428.705903:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626428.705906:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626428.705909:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626428.705918:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828388608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626428.705924:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626428.705926:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626428.705927:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626428.705935:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10763 00000100:00080000:1.0:1778626428.705946:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626428.705949:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626428.705953:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626428.705955:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626428.705964:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828388736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626428.705973:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626428.705975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626428.705976:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626428.705980:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10763 00000100:00080000:1.0:1778626428.705983:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626428.705985:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626428.705989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626428.705992:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626428/real 1778626428] req@ffff954350195880 x1865013828388352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626428.705997:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828388864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626428.706039:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828388352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626428.706052:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626428.706056:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626428.706059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828388352:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626428.706090:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626428.706109:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626428.706129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626428/real 1778626428] req@ffff954350197b80 x1865013828388480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626428.706138:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828388480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626428.706144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626428.706146:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626428.706148:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626428.706152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388608:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626428.706162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626428.706168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626428/real 1778626428] req@ffff954350196a00 x1865013828388608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626428.706175:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828388608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626428.706189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626428.706191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626428.706194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388608:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626428.706198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626428.706211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626428.706217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626428/real 1778626428] req@ffff954350196680 x1865013828388736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626428.706224:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828388736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626428.706229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626428.706231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626428.706233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626428.706236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388864:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626428.706245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626428.706250:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626428/real 1778626428] req@ffff954350194e00 x1865013828388864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626428.706258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828388864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626428.706263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626428.706264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626428.706270:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828388864:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626428.897827:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626429.665791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626429.921867:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626430.689749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626430.945800:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626431.713750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626431.969798:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626432.737746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626432.993818:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626433.761737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626433.825751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626433.825760:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626433.825765:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626433.825775:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10768 00000100:00080000:3.0:1778626433.825787:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626433.825794:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626433.825830:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828388992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626433.825862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626433.825864:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626433.825866:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626433.825870:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10768 00000100:00080000:3.0:1778626433.825873:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626433.825876:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626433.825883:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626433.825894:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828389120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626433.825902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626433.825906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778626433.825907:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828388992:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626433.825908:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626433.825915:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10768 00000100:00080000:3.0:1778626433.825917:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626433.825919:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626433.825937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626433.825947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828389248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626433.825954:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626433.825957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626433.825958:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626433.825960:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10768 00000100:00080000:3.0:1778626433.825963:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778626433.825965:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626433.825969:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626433.825985:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626433.825996:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701fb80 x1865013828389376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626433.826003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626433.826006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626433.826008:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626433.826011:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10768 00000100:00100000:0.0:1778626433.826015:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626433/real 1778626433] req@ffff95434701f480 x1865013828388992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626433.826015:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626433.826022:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626433.826028:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626433.826038:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828389504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626433.826041:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828388992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626433.826064:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626433.826069:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626433.826074:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828388992:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626433.826099:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626433.826123:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626433.826133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626433/real 1778626433] req@ffff95434701e680 x1865013828389120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626433.826142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828389120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626433.826147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626433.826150:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626433.826158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626433.826162:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389248:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626433.826177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626433.826188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626433/real 1778626433] req@ffff95434701d180 x1865013828389248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626433.826200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828389248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626433.826208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626433.826211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626433.826215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389248:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626433.826222:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626433.826241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626433.826249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626433/real 1778626433] req@ffff95434701fb80 x1865013828389376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626433.826257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701fb80 x1865013828389376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626433.826262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626433.826264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626433.826266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626433.826270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389504:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626433.826280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626433.826291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626433/real 1778626433] req@ffff95434701ca80 x1865013828389504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626433.826300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828389504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626433.826309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626433.826312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626433.826315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389504:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626434.017805:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626434.785772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626435.041833:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626435.809758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626436.065814:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626436.833741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626437.089848:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626437.857753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626438.113823:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626438.881733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626438.945764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626438.945771:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626438.945775:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626438.945783:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10773 00000100:00080000:3.0:1778626438.945793:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626438.945799:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626438.945830:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828389632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626438.945854:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626438.945856:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626438.945861:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626438.945864:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10773 00000100:00080000:3.0:1778626438.945867:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626438.945869:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626438.945889:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626438.945900:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828389760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626438.945908:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626438.945910:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626438.945911:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626438.945914:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10773 00000100:00080000:3.0:1778626438.945917:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626438.945922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828389632:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626438.945922:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626438.945926:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626438.945955:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828389888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626438.945963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626438.945966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626438.945967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626438.945970:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10773 00000100:00080000:3.0:1778626438.945973:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626438.945975:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626438.945979:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626438.945987:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828390016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626438.945993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626438.945995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626438.945996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626438.945999:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10773 00000100:00080000:3.0:1778626438.946001:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626438.946003:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626438.946007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626438.946015:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828390144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626438.946017:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626438.946072:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389888:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626438.946100:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626438.946127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390144:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626439.137902:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626439.905823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626440.161843:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626440.929746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626441.185840:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626441.953778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778626442.017931:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778626442.017970:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778626442.017986:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626442.018050:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626442.018059:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626442.018067:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626442.018075:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626442.018155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626438/real 1778626442] req@ffff95434701df80 x1865013828389888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626442.018181:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828389888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626442.018210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626442.018216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626442.018220:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389888:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626442.018229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626438/real 1778626442] req@ffff95434701f800 x1865013828390016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626442.018244:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828390016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626442.018252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626442.018257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626442.018260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626442.018267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626438/real 1778626442] req@ffff95434701d880 x1865013828390144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626442.018280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828390144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626442.018288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626442.018292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626442.018296:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390144:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626442.018302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626438/real 1778626442] req@ffff95434701ea00 x1865013828389632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626442.018315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828389632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626442.018331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626442.018335:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626442.018339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828389632:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626442.018345:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626438/real 1778626442] req@ffff95434701c000 x1865013828389760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626442.018357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828389760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626442.018365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626442.018369:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626442.018371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828389760:192.168.204.102@tcp:38:kworker.0 00010000:00010000:1.0:1778626442.209833:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626442.977780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626443.233827:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626444.001768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626444.065744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626444.065781:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626444.065786:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626444.065796:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10778 00000100:00080000:3.0:1778626444.065806:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626444.065813:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626444.065844:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828390272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626444.065870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626444.065872:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626444.065874:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626444.065877:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10778 00000100:00080000:3.0:1778626444.065880:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626444.065884:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626444.065893:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626444.065908:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828390400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626444.065920:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626444.065925:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828390272:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626444.065926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626444.065930:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626444.065939:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10778 00000100:00080000:3.0:1778626444.065943:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626444.065947:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626444.065953:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626444.065967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828390528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626444.065987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626444.065991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626444.065996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778626444.065997:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626444.066006:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10778 00000100:00080000:3.0:1778626444.066011:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626444.066016:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626444.066022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626444.066036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828390656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626444.066040:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626444/real 1778626444] req@ffff95434701c380 x1865013828390272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626444.066059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626444.066064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626444.066066:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626444.066071:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10778 00000100:00100000:0.0:1778626444.066075:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828390272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626444.066077:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626444.066086:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626444.066096:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778626444.066098:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626444.066106:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626444.066112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828390272:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626444.066114:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828390784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626444.066160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626444.066186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626444.066199:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626444/real 1778626444] req@ffff95434701f100 x1865013828390400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626444.066213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828390400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626444.066222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626444.066226:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626444.066230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626444.066237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390528:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626444.066255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626444.066271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626444/real 1778626444] req@ffff95434701c700 x1865013828390528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626444.066296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828390528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626444.066306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626444.066310:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626444.066314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390528:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626444.066321:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626444.066341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626444.066352:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626444/real 1778626444] req@ffff95434701d500 x1865013828390656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626444.066372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828390656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626444.066382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626444.066385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626444.066389:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626444.066401:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390784:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626444.066421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626444.066431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626444/real 1778626444] req@ffff95434701c000 x1865013828390784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626444.066444:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828390784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626444.066453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626444.066456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626444.066460:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828390784:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626444.257858:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626445.025733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626445.281814:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626446.049807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626446.305918:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626447.073751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626447.329810:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626448.097764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626448.353837:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626449.121790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626449.185762:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626449.185771:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626449.185776:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626449.185786:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10783 00000100:00080000:2.0:1778626449.185797:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626449.185822:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626449.185853:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828390912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626449.185876:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626449.185879:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626449.185880:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626449.185883:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10783 00000100:00080000:2.0:1778626449.185886:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626449.185889:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626449.185897:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626449.185905:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828391040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626449.185914:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626449.185917:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828390912:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626449.185917:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626449.185920:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626449.185926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10783 00000100:00080000:2.0:1778626449.185928:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626449.185931:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626449.185933:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626449.185942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788000 x1865013828391168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626449.185949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626449.185952:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626449.185959:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626449.185962:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10783 00000100:00080000:2.0:1778626449.185976:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626449.185979:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626449.185983:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626449.185987:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626449.185995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828391296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626449.186002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626449.186005:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626449.186006:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626449.186008:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10783 00000100:00080000:2.0:1778626449.186011:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626449.186014:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626449.186018:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626449.186022:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626449/real 1778626449] req@ffff954358789c00 x1865013828390912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626449.186031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828391424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626449.186068:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828390912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626449.186077:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626449.186080:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626449.186082:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828390912:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626449.186100:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626449.186117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626449.186125:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626449/real 1778626449] req@ffff954358789180 x1865013828391040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626449.186134:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828391040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626449.186140:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626449.186142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626449.186144:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626449.186147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391168:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626449.186157:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626449.186162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626449/real 1778626449] req@ffff954358788000 x1865013828391168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626449.186169:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828391168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626449.186183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626449.186185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626449.186187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391168:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626449.186192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626449.186205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626449.186210:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626449/real 1778626449] req@ffff954358788e00 x1865013828391296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626449.186217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828391296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626449.186222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626449.186224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626449.186225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626449.186229:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391424:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626449.186238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626449.186243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626449/real 1778626449] req@ffff954358789500 x1865013828391424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626449.186250:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828391424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626449.186255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626449.186257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626449.186263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391424:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626449.377804:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626450.145820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626450.401820:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626451.169735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626451.425807:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626452.193777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626452.449832:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626453.217781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626453.473780:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626454.241780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626454.305787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626454.305796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626454.305799:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626454.305807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10788 00000100:00080000:3.0:1778626454.305817:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626454.305823:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626454.305855:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828391552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626454.305877:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626454.305880:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626454.305881:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626454.305885:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10788 00000100:00080000:3.0:1778626454.305887:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626454.305889:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626454.305896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626454.305905:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828391680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626454.305913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626454.305923:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828391552:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626454.305923:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626454.305925:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626454.305932:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10788 00000100:00080000:3.0:1778626454.305935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626454.305937:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626454.305941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626454.305966:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828391808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626454.305973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626454.305975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626454.305977:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626454.305980:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10788 00000100:00080000:3.0:1778626454.305983:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626454.305989:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626454.305992:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778626454.305994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626454.306004:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828391936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626454.306011:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626454.306013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626454.306015:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626454.306017:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10788 00000100:00080000:3.0:1778626454.306020:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626454.306023:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626454.306027:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626454.306033:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626454/real 1778626454] req@ffff95434701ea00 x1865013828391552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626454.306036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828392064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626454.306080:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828391552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626454.306092:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626454.306097:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626454.306101:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828391552:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626454.306124:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626454.306146:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626454.306159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626454/real 1778626454] req@ffff95434701d880 x1865013828391680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626454.306171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828391680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626454.306180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626454.306184:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626454.306194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626454.306201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391808:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626454.306220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626454.306230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626454/real 1778626454] req@ffff95434701f800 x1865013828391808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626454.306242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828391808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626454.306250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626454.306253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626454.306256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391808:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626454.306262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626454.306280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626454.306289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626454/real 1778626454] req@ffff95434701df80 x1865013828391936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626454.306300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828391936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626454.306307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626454.306311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626454.306314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828391936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626454.306320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392064:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626454.306340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626454.306360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626454/real 1778626454] req@ffff95434701ca80 x1865013828392064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626454.306373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828392064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626454.306382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626454.306385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626454.306388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392064:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626454.497790:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626455.265768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626455.521812:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626456.289748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626456.545826:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626457.313750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626457.569843:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626458.337748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626458.593862:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626459.361810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626459.425819:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626459.425833:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626459.425839:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626459.425851:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10793 00000100:00080000:2.0:1778626459.425866:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626459.425873:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626459.425916:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828392192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626459.425964:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626459.425967:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626459.425968:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626459.425972:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10793 00000100:00080000:2.0:1778626459.425975:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626459.425979:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626459.425989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626459.426000:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828392320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626459.426009:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828392192:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626459.426010:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626459.426017:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626459.426020:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626459.426024:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10793 00000100:00080000:2.0:1778626459.426026:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626459.426028:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626459.426032:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626459.426044:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828392448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626459.426054:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626459.426058:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626459.426060:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626459.426065:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10793 00000100:00080000:2.0:1778626459.426078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626459.426084:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626459.426090:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626459.426099:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626459.426105:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828392576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626459.426116:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626459.426119:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626459.426121:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626459.426124:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10793 00000100:00080000:2.0:1778626459.426127:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626459.426130:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626459.426134:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626459.426144:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626459/real 1778626459] req@ffff954358788a80 x1865013828392192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626459.426145:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828392704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626459.426186:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828392192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626459.426196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626459.426199:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626459.426202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828392192:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626459.426225:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626459.426241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626459.426258:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626459/real 1778626459] req@ffff954358789880 x1865013828392320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626459.426267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828392320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626459.426273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626459.426275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626459.426278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626459.426281:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392448:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626459.426292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626459.426298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626459/real 1778626459] req@ffff954358788700 x1865013828392448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626459.426305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828392448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626459.426310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626459.426311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626459.426313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392448:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626459.426317:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626459.426327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626459.426333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626459/real 1778626459] req@ffff95435878bb80 x1865013828392576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626459.426340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828392576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626459.426351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626459.426353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626459.426355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626459.426359:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392704:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626459.426372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626459.426378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626459/real 1778626459] req@ffff954358788380 x1865013828392704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626459.426385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828392704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626459.426391:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626459.426393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626459.426396:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392704:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626459.617798:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626460.385746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626460.641828:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626461.409789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626461.665803:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626462.433734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626462.689830:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626463.457802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626463.713794:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626464.481743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626464.545772:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626464.545782:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626464.545787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626464.545796:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10799 00000100:00080000:1.0:1778626464.545807:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626464.545813:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626464.545845:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828392832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626464.545867:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626464.545870:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626464.545871:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626464.545874:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10799 00000100:00080000:1.0:1778626464.545876:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626464.545878:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626464.545885:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626464.545896:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828392960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626464.545906:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626464.545910:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778626464.545921:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828392832:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626464.545927:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626464.545933:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10799 00000100:00080000:1.0:1778626464.545938:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626464.545942:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626464.545949:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626464.545962:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828393088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626464.545969:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626464.545972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626464.545973:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626464.545976:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10799 00000100:00080000:1.0:1778626464.545979:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626464.545982:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626464.545985:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626464.545992:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626464.545994:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828393216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626464.546001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626464.546003:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626464.546004:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626464.546008:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10799 00000100:00080000:1.0:1778626464.546016:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626464.546020:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626464.546025:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626464.546034:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626464/real 1778626464] req@ffff954350195f80 x1865013828392832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626464.546046:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828393344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626464.546067:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828392832/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626464.546087:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626464.546093:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626464.546098:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828392832:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626464.546106:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626464.546130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626464.546143:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626464/real 1778626464] req@ffff954350195180 x1865013828392960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626464.546156:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828392960/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626464.546165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626464.546169:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626464.546172:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828392960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626464.546196:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393088:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626464.546215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626464.546226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626464/real 1778626464] req@ffff954350194e00 x1865013828393088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626464.546238:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828393088/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626464.546247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626464.546250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626464.546254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393088:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626464.546266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626464.546282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626464.546290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626464/real 1778626464] req@ffff954350196680 x1865013828393216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626464.546298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828393216/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626464.546303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626464.546305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626464.546307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626464.546311:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393344:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626464.546320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626464.546325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626464/real 1778626464] req@ffff954350196a00 x1865013828393344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626464.546331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828393344/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626464.546337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626464.546338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626464.546340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393344:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626464.737849:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626465.505746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626465.761808:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626466.529733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626466.785825:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626467.553773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626467.809836:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626468.577780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626468.833825:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626469.601772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626469.665768:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626469.665776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626469.665780:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626469.665788:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10804 00000100:00080000:1.0:1778626469.665797:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626469.665809:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626469.665842:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828393472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626469.665865:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626469.665867:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626469.665869:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626469.665872:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10804 00000100:00080000:1.0:1778626469.665874:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626469.665876:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626469.665884:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626469.665892:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828393600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626469.665899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626469.665901:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626469.665916:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626469.665919:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10804 00000100:00080000:1.0:1778626469.665922:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626469.665925:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626469.665929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626469.665937:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828393728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626469.665944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626469.665947:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1778626469.665950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828393472:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626469.665950:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626469.665957:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10804 00000100:00080000:1.0:1778626469.665959:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626469.665962:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626469.665965:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626469.665974:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828393856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626469.665979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626469.665982:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626469.665983:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626469.665986:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10804 00000100:00080000:1.0:1778626469.665987:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626469.665990:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626469.665992:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626469.666004:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828393984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626469.666020:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626469.666054:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626469/real 1778626469] req@ffff954350197b80 x1865013828393472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626469.666069:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828393472/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626469.666076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626469.666079:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626469.666082:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828393472:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626469.666088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626469.666102:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626469.666120:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626469/real 1778626469] req@ffff954350195880 x1865013828393600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626469.666127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828393600/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626469.666132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626469.666134:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626469.666136:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626469.666153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393728:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626469.666163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626469.666168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626469/real 1778626469] req@ffff954350194380 x1865013828393728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626469.666183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828393728/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626469.666189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626469.666191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626469.666193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393728:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626469.666196:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626469.666208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626469.666214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626469/real 1778626469] req@ffff954350197800 x1865013828393856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626469.666221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828393856/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626469.666226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626469.666237:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626469.666238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626469.666242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393984:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626469.666250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626469.666254:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626469/real 1778626469] req@ffff954350195c00 x1865013828393984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626469.666261:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828393984/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626469.666288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626469.666290:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626469.666292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828393984:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626469.857864:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626470.625776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626470.882168:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626471.649738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626471.905963:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626472.673724:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626472.929826:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626473.697769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626473.953959:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626474.721753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626474.785732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626474.785741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626474.785744:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626474.785752:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10809 00000100:00080000:1.0:1778626474.785763:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626474.785769:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626474.785799:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828394112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626474.785822:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626474.785825:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626474.785826:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626474.785829:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10809 00000100:00080000:1.0:1778626474.785832:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626474.785834:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626474.785842:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626474.785851:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828394240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626474.785859:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626474.785864:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778626474.785864:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828394112:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626474.785867:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626474.785884:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10809 00000100:00080000:1.0:1778626474.785888:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626474.785892:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626474.785898:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626474.785912:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828394368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626474.785924:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626474.785925:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626474.785929:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626474.785931:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626474.785938:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10809 00000100:00080000:1.0:1778626474.785942:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626474.785948:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626474.785954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626474.785955:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626474/real 1778626474] req@ffff954350195500 x1865013828394112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626474.785972:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828394496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626474.785976:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828394112/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626474.786006:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778626474.786007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626474.786011:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778626474.786012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626474.786015:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626474.786016:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828394112:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626474.786019:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10809 00000100:00080000:1.0:1778626474.786022:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626474.786037:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626474.786043:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626474.786051:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778626474.786056:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828394624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626474.786085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626474.786099:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626474/real 1778626474] req@ffff954350194a80 x1865013828394240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626474.786110:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828394240/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626474.786124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626474.786128:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626474.786131:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626474.786138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394368:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626474.786159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626474.786184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626474/real 1778626474] req@ffff954350194700 x1865013828394368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626474.786196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828394368/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626474.786205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626474.786208:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626474.786211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394368:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626474.786218:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626474.786235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626474.786242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626474/real 1778626474] req@ffff954350197480 x1865013828394496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626474.786257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828394496/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626474.786263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626474.786265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626474.786267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626474.786280:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394624:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626474.786292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626474.786298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626474/real 1778626474] req@ffff954350196d80 x1865013828394624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626474.786305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828394624/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626474.786311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626474.786312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626474.786314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394624:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626474.977887:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626475.745775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626476.001856:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626476.769734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626477.025823:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626477.793815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626478.049794:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626478.817735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626479.073815:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626479.841767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626479.905789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626479.905800:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626479.905804:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626479.905815:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10814 00000100:00080000:0.0:1778626479.905829:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626479.905837:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626479.905905:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828394752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626479.905957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626479.905961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626479.905962:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626479.905966:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10814 00000100:00080000:0.0:1778626479.905986:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626479.905991:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626479.906001:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626479.906016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828394880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626479.906016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828394752:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626479.906030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626479.906033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626479.906034:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626479.906037:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10814 00000100:00080000:0.0:1778626479.906039:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626479.906042:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626479.906045:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626479.906061:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828395008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626479.906080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626479.906083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626479.906084:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626479.906088:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10814 00000100:00080000:0.0:1778626479.906092:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626479.906094:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626479.906098:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626479.906107:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828395136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626479.906122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626479.906130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626479.906132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626479.906134:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626479.906138:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10814 00000100:00080000:0.0:1778626479.906142:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626479.906147:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626479.906153:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626479.906169:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828395264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626479.906169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626479/real 1778626479] req@ffff954342f4a680 x1865013828394752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626479.906218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828394752/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626479.906241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626479.906244:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626479.906248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828394752:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626479.906273:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626479.906293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626479.906302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626479/real 1778626479] req@ffff954342f48700 x1865013828394880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626479.906310:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828394880/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626479.906316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626479.906318:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626479.906321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828394880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626479.906324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395008:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626479.906334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626479.906339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626479/real 1778626479] req@ffff954342f49f80 x1865013828395008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626479.906346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828395008/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626479.906352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626479.906354:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626479.906356:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395008:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626479.906365:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626479.906379:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626479.906386:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626479/real 1778626479] req@ffff954342f4b100 x1865013828395136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626479.906394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828395136/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626479.906399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626479.906401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626479.906402:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626479.906406:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395264:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626479.906416:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626479.906421:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626479/real 1778626479] req@ffff954342f48a80 x1865013828395264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626479.906428:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828395264/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626479.906433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626479.906435:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626479.906437:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395264:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626480.097945:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626480.865842:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626481.121859:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626481.889763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626482.145868:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626482.913778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626483.169842:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626483.937737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626484.193872:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626484.961780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626485.025751:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626485.025760:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626485.025763:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626485.025785:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10819 00000100:00080000:2.0:1778626485.025794:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626485.025799:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626485.025828:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828395392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626485.025861:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626485.025864:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626485.025865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626485.025869:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10819 00000100:00080000:2.0:1778626485.025871:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626485.025874:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626485.025881:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626485.025890:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828395520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626485.025898:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626485.025900:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626485.025902:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626485.025905:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10819 00000100:00080000:2.0:1778626485.025909:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626485.025913:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626485.025918:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626485.025922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828395392:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626485.025932:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828395648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626485.025958:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626485.025974:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626485.025976:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626485.025980:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10819 00000100:00080000:2.0:1778626485.025984:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626485.025987:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626485.025992:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626485.026000:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626485.026004:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828395776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626485.026012:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626485.026015:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626485.026016:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626485.026019:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10819 00000100:00080000:2.0:1778626485.026022:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626485.026024:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626485.026027:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626485.026038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828395904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626485.026039:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626485/real 1778626485] req@ffff95435878b100 x1865013828395392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626485.026070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828395392/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626485.026081:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626485.026085:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626485.026089:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828395392:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626485.026105:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626485.026128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626485.026140:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626485/real 1778626485] req@ffff95435878b480 x1865013828395520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626485.026151:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828395520/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626485.026160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626485.026163:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626485.026167:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626485.026191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395648:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626485.026208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626485.026218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626485/real 1778626485] req@ffff95435878aa00 x1865013828395648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626485.026229:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828395648/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626485.026237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626485.026240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626485.026242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395648:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626485.026249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626485.026264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626485.026273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626485/real 1778626485] req@ffff954358788380 x1865013828395776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626485.026291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828395776/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626485.026298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626485.026301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626485.026305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626485.026310:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395904:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626485.026327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626485.026336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626485/real 1778626485] req@ffff95435878bb80 x1865013828395904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626485.026346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828395904/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626485.026354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626485.026357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626485.026360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828395904:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626485.217842:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626485.985762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626486.241836:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626487.009763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626487.265855:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626488.033755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626488.289830:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626489.057748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626489.313841:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626490.081739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626490.145737:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626490.145744:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626490.145748:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626490.145756:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10824 00000100:00080000:2.0:1778626490.145777:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626490.145782:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626490.145811:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828396032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626490.145835:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626490.145838:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626490.145841:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626490.145845:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10824 00000100:00080000:2.0:1778626490.145848:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626490.145852:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626490.145873:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626490.145886:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828396160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626490.145896:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626490.145900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828396032:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626490.145900:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626490.145901:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626490.145905:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10824 00000100:00080000:2.0:1778626490.145908:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626490.145912:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626490.145916:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626490.145925:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828396288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626490.145934:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626490.145942:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626490.145944:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626490.145948:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10824 00000100:00080000:2.0:1778626490.145951:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626490.145955:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626490.145957:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778626490.145960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626490.145969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828396416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626490.145978:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626490.145981:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626490.145983:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626490.145986:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10824 00000100:00100000:0.0:1778626490.145987:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626490/real 1778626490] req@ffff954358788700 x1865013828396032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626490.145989:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626490.145992:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626490.145996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626490.146004:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828396544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626490.146012:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828396032/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626490.146022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626490.146027:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626490.146031:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828396032:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626490.146053:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626490.146077:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626490.146091:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626490/real 1778626490] req@ffff954358789880 x1865013828396160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626490.146105:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828396160/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626490.146113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626490.146117:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626490.146120:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626490.146126:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396288:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626490.146148:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626490.146159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626490/real 1778626490] req@ffff954358788a80 x1865013828396288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626490.146172:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828396288/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626490.146200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626490.146202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626490.146205:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396288:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626490.146209:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626490.146225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626490.146233:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626490/real 1778626490] req@ffff954358789500 x1865013828396416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626490.146240:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828396416/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626490.146246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626490.146248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626490.146250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626490.146253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396544:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626490.146261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626490.146266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626490/real 1778626490] req@ffff954358788e00 x1865013828396544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626490.146273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828396544/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626490.146278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626490.146281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626490.146290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396544:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626490.337797:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626491.105754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626491.361834:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626492.129762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626492.385816:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626493.153816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626493.409853:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626494.177739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626494.433872:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626495.201784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626495.265832:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626495.265840:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626495.265844:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626495.265852:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10829 00000100:00080000:1.0:1778626495.265861:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626495.265867:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626495.265904:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828396672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626495.265958:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626495.265963:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626495.265965:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626495.265974:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10829 00000100:00080000:1.0:1778626495.265978:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626495.265982:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626495.265992:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626495.266007:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828396800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626495.266012:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828396672:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626495.266017:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626495.266021:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626495.266022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626495.266035:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10829 00000100:00080000:1.0:1778626495.266039:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626495.266043:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626495.266047:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626495.266057:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828396928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626495.266066:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626495.266069:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626495.266071:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626495.266074:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10829 00000800:00000100:0.0:1778626495.266080:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626495.266083:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626495.266087:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626495.266092:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626495.266102:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828397056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626495.266110:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626495.266114:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626495.266115:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626495.266116:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626495/real 1778626495] req@ffff954350194000 x1865013828396672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626495.266119:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10829 00000100:00080000:1.0:1778626495.266123:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626495.266126:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626495.266130:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626495.266133:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828396672/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626495.266138:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828397184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626495.266151:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626495.266155:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626495.266159:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828396672:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626495.266184:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626495.266209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626495.266223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626495/real 1778626495] req@ffff954350197100 x1865013828396800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626495.266236:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828396800/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626495.266245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626495.266249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626495.266260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626495.266265:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396928:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626495.266282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626495.266298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626495/real 1778626495] req@ffff954350196d80 x1865013828396928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626495.266306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828396928/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626495.266311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626495.266313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626495.266315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828396928:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626495.266319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626495.266331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626495.266337:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626495/real 1778626495] req@ffff954350197480 x1865013828397056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626495.266344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828397056/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626495.266349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626495.266351:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626495.266352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626495.266356:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397184:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626495.266365:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626495.266377:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626495/real 1778626495] req@ffff954350194700 x1865013828397184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626495.266384:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828397184/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626495.266390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626495.266392:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626495.266394:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397184:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626495.457847:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626496.225765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626496.481809:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626497.249783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626497.505853:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626498.273861:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626498.529951:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626499.297801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626499.553845:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626500.321802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626500.385783:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626500.385791:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626500.385796:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626500.385806:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10834 00000100:00080000:1.0:1778626500.385818:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626500.385826:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626500.385875:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828397312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626500.385900:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626500.385904:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626500.385906:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626500.385911:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10834 00000100:00080000:1.0:1778626500.385917:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626500.385921:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626500.385925:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828397312:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778626500.385933:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626500.385950:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828397440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626500.385962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626500.385966:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626500.385969:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626500.385973:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10834 00000100:00080000:1.0:1778626500.385977:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778626500.385982:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626500.385985:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626500.385992:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626500.386003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828397568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626500.386014:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626500.386018:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626500.386020:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626500.386023:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626500/real 1778626500] req@ffff954350194a80 x1865013828397312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626500.386036:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10834 00000100:00080000:1.0:1778626500.386042:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626500.386046:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828397312/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626500.386050:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626500.386061:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778626500.386062:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:0.0:1778626500.386071:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626500.386077:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828397312:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626500.386081:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828397696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626500.386094:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626500.386098:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626500.386100:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626500.386105:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10834 00000100:00100000:0.0:1778626500.386109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397440:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778626500.386111:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626500.386119:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626500.386125:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626500.386138:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828397824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626500.386142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626500.386166:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626500/real 1778626500] req@ffff954350195500 x1865013828397440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626500.386179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828397440/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626500.386188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626500.386192:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626500.386196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626500.386202:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397568:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626500.386224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626500.386235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626500/real 1778626500] req@ffff954350195c00 x1865013828397568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626500.386244:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828397568/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626500.386249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626500.386251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626500.386254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397568:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626500.386258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626500.386271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626500.386277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626500/real 1778626500] req@ffff954350197800 x1865013828397696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626500.386284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828397696/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626500.386294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626500.386297:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626500.386299:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626500.386311:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397824:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626500.386324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626500.386330:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626500/real 1778626500] req@ffff954350194380 x1865013828397824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626500.386337:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828397824/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626500.386342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626500.386344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626500.386346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828397824:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626500.577801:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626501.345754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626501.601825:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626502.369769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626502.625808:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626503.393736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626503.649812:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626504.417743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626504.673848:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626505.441735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626505.505754:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626505.505765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626505.505769:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626505.505780:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10840 00000100:00080000:1.0:1778626505.505792:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626505.505798:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626505.505843:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828397952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626505.505868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626505.505871:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626505.505872:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626505.505875:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10840 00000100:00080000:1.0:1778626505.505878:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626505.505881:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626505.505888:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626505.505897:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828398080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626505.505904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626505.505907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626505.505908:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626505.505910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10840 00000100:00080000:1.0:1778626505.505912:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626505.505917:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626505.505919:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828397952:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778626505.505921:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626505.505932:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828398208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626505.505939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626505.505941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626505.505942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626505.505950:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10840 00000100:00080000:1.0:1778626505.505953:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626505.505956:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626505.505960:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626505.505968:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828398336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626505.505975:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626505.505977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626505.505979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626505.505981:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10840 00000100:00080000:1.0:1778626505.505983:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626505.505985:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626505.505988:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626505.505995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828398464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626505.506004:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626505.506034:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398208:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626505.506048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626505.506073:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398464:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626505.697818:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626506.465789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626506.721845:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626507.489740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626507.745817:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626508.513742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778626508.577971:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778626508.578020:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778626508.578042:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626508.578097:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626508.578109:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626508.578118:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626508.578129:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626508.578158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626505/real 1778626508] req@ffff954350196a00 x1865013828398208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626508.578191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828398208/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626508.578205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626508.578212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626508.578217:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398208:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626508.578227:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626505/real 1778626508] req@ffff954350196680 x1865013828398336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626508.578239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828398336/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626508.578247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626508.578252:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626508.578255:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626508.578263:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626505/real 1778626508] req@ffff954350194e00 x1865013828398464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626508.578275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828398464/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626508.578310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626508.578320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626508.578324:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398464:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626508.578332:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626505/real 1778626508] req@ffff954350195880 x1865013828397952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626508.578347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828397952/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626508.578355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626508.578358:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626508.578361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828397952:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626508.578369:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626505/real 1778626508] req@ffff954350197b80 x1865013828398080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626508.578382:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828398080/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626508.578391:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626508.578395:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626508.578398:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398080:192.168.204.102@tcp:38:kworker.0 00010000:00010000:3.0:1778626508.769873:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626509.537734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626509.793848:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626510.561760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626510.625802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626510.625811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626510.625827:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626510.625835:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10845 00000100:00080000:0.0:1778626510.625844:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626510.625850:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626510.625879:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828398592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626510.625904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626510.625907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626510.625908:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626510.625911:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10845 00000100:00080000:0.0:1778626510.625913:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626510.625915:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626510.625923:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626510.625930:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828398720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626510.625937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626510.625940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626510.625942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626510.625944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10845 00000100:00080000:0.0:1778626510.625946:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626510.625948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626510.625953:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626510.625963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828398848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626510.625974:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828398592:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626510.625994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626510.625997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626510.625999:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626510.626007:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10845 00000100:00080000:0.0:1778626510.626016:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626510.626021:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626510.626028:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626510.626041:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828398976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626510.626047:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626510.626054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626510.626058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626510.626060:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626510.626065:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10845 00000100:00080000:0.0:1778626510.626069:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626510.626074:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626510.626080:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626510.626089:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626510/real 1778626510] req@ffff954342f4ad80 x1865013828398592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626510.626093:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828399104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626510.626127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828398592/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626510.626146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626510.626151:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626510.626156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828398592:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626510.626164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626510.626188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626510.626204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626510/real 1778626510] req@ffff954342f49500 x1865013828398720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626510.626217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828398720/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626510.626226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626510.626229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626510.626232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626510.626258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398848:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626510.626279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626510.626291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626510/real 1778626510] req@ffff954342f49c00 x1865013828398848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626510.626303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828398848/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626510.626311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626510.626315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626510.626319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398848:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626510.626332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626510.626355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626510.626367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626510/real 1778626510] req@ffff954342f4b480 x1865013828398976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626510.626380:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828398976/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626510.626388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626510.626391:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626510.626394:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828398976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626510.626400:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399104:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626510.626420:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626510.626431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626510/real 1778626510] req@ffff954342f4a300 x1865013828399104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626510.626443:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828399104/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626510.626451:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626510.626454:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626510.626473:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399104:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626510.817857:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626511.585758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626511.841847:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626512.609730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626512.865838:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626513.633758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626513.889860:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626514.657807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626514.913868:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626515.681749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626515.745761:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626515.745768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626515.745771:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626515.745779:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10850 00000100:00080000:0.0:1778626515.745788:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626515.745793:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626515.745824:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828399232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626515.745850:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626515.745854:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626515.745856:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626515.745861:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10850 00000100:00080000:0.0:1778626515.745864:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626515.745868:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626515.745877:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626515.745891:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828399360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626515.745904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626515.745907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626515.745909:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626515.745912:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10850 00000100:00080000:0.0:1778626515.745914:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626515.745917:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626515.745921:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626515.745938:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828399232:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626515.745948:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828399488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626515.745957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626515.745960:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626515.745962:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626515.745966:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10850 00000100:00080000:0.0:1778626515.745972:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626515.745975:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626515.745979:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626515.745986:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828399616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626515.745992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626515.745994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626515.745996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626515.746001:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10850 00000100:00080000:0.0:1778626515.746003:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626515.746005:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626515.746008:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626515.746014:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828399744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626515.746030:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626515.746074:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626515/real 1778626515] req@ffff954342f49880 x1865013828399232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626515.746115:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828399232/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626515.746125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626515.746129:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626515.746132:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828399232:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626515.746139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626515.746159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626515.746168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626515/real 1778626515] req@ffff954342f4bb80 x1865013828399360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626515.746177:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828399360/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626515.746182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626515.746185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626515.746187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626515.746211:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399488:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626515.746221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626515.746227:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626515/real 1778626515] req@ffff954342f49180 x1865013828399488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626515.746235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828399488/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626515.746249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626515.746252:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626515.746254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399488:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626515.746258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626515.746273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626515.746279:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626515/real 1778626515] req@ffff954342f48000 x1865013828399616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626515.746287:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828399616/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626515.746292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626515.746294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626515.746295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626515.746299:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399744:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626515.746317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626515.746327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626515/real 1778626515] req@ffff954342f48380 x1865013828399744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626515.746340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828399744/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626515.746348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626515.746350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626515.746378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828399744:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626515.937881:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626516.705837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626516.961852:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626517.729739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626517.985832:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626518.753738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626519.009819:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626519.777764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626520.033848:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626520.801749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626520.865814:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626520.865823:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626520.865843:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626520.865851:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10855 00000100:00080000:1.0:1778626520.865860:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626520.865866:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626520.865895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828399872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626520.865919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626520.865922:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626520.865924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626520.865927:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10855 00000100:00080000:1.0:1778626520.865929:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626520.865931:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626520.865938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626520.865945:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828400000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626520.865953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626520.865955:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626520.865957:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626520.865959:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10855 00000100:00080000:1.0:1778626520.865964:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626520.865966:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626520.865970:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626520.865978:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828400128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626520.865996:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626520.865999:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626520.866006:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626520.866008:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828399872:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626520.866016:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10855 00000100:00080000:1.0:1778626520.866023:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626520.866028:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626520.866035:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626520.866052:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828400256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626520.866063:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626520.866067:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626520.866069:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626520.866073:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10855 00000100:00080000:1.0:1778626520.866078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626520.866081:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626520.866086:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626520.866098:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828400384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626520.866103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626520.866161:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626520/real 1778626520] req@ffff954350195180 x1865013828399872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626520.866187:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828399872/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626520.866197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626520.866202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626520.866207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828399872:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626520.866217:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626520.866247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626520.866323:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626520/real 1778626520] req@ffff954350195f80 x1865013828400000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626520.866335:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828400000/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626520.866342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626520.866344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626520.866347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626520.866373:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400128:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626520.866390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626520.866412:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626520/real 1778626520] req@ffff954350197b80 x1865013828400128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626520.866421:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828400128/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626520.866426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626520.866429:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626520.866431:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400128:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626520.866436:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626520.866449:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626520.866455:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626520/real 1778626520] req@ffff954350195880 x1865013828400256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626520.866463:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828400256/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626520.866468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626520.866470:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626520.866471:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626520.866476:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400384:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626520.866486:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626520.866491:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626520/real 1778626520] req@ffff954350194e00 x1865013828400384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626520.866499:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828400384/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626520.866509:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626520.866512:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626520.866514:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400384:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626521.057829:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626521.825772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626522.081809:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626522.849744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626523.105829:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626523.873774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626524.129942:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626524.897728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626525.153824:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626525.921819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626525.985788:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626525.985802:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626525.985832:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626525.985844:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10860 00000100:00080000:1.0:1778626525.985860:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626525.985869:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626525.985915:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828400512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626525.985952:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626525.985956:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626525.985958:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626525.985963:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10860 00000100:00080000:1.0:1778626525.985968:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626525.985972:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626525.986000:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626525.986016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828400640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626525.986029:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626525.986033:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626525.986036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626525.986041:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10860 00000100:00080000:1.0:1778626525.986046:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626525.986050:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626525.986072:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626525.986077:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828400512:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626525.986108:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828400768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626525.986136:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626525.986140:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626525.986142:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626525.986148:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10860 00000100:00080000:1.0:1778626525.986154:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626525.986171:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626525.986178:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626525.986182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626525.986214:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828400896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626525.986226:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626525.986231:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626525.986233:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626525.986239:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10860 00000100:00080000:1.0:1778626525.986244:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626525.986247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626525/real 1778626525] req@ffff954350196680 x1865013828400512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626525.986253:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626525.986264:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626525.986276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828400512/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626525.986280:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828401024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626525.986294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626525.986301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626525.986304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828400512:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626525.986312:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626525.986332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626525.986342:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626525/real 1778626525] req@ffff954350196a00 x1865013828400640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626525.986350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828400640/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626525.986356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626525.986359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626525.986361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626525.986411:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400768:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626525.986427:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626525.986435:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626525/real 1778626525] req@ffff954350194380 x1865013828400768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626525.986444:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828400768/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626525.986466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626525.986468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626525.986470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400768:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626525.986474:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626525.986486:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626525.986491:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626525/real 1778626525] req@ffff954350197800 x1865013828400896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626525.986498:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828400896/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626525.986504:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626525.986505:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626525.986507:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828400896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626525.986510:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401024:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626525.986519:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626525.986524:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626525/real 1778626525] req@ffff954350195c00 x1865013828401024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626525.986538:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828401024/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626525.986544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626525.986546:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626525.986548:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401024:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626526.177845:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626526.945740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626527.201831:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626527.969766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626528.225819:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626528.993737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626529.249850:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626530.017775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626530.273802:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626531.041751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626531.105758:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626531.105766:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626531.105770:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626531.105777:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10865 00000100:00080000:1.0:1778626531.105802:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626531.105807:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626531.105845:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828401152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626531.105868:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626531.105870:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626531.105872:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626531.105875:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10865 00000100:00080000:1.0:1778626531.105877:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626531.105879:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626531.105886:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626531.105894:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828401280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626531.105902:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626531.105904:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626531.105906:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626531.105908:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10865 00000100:00080000:1.0:1778626531.105910:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626531.105913:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626531.105917:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626531.105926:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828401152:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626531.105930:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828401408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626531.105942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626531.105952:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626531.105955:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626531.105960:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10865 00000100:00080000:1.0:1778626531.105966:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626531.105971:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626531.105977:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626531.105984:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626531.105991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828401536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626531.105999:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626531.106002:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626531.106003:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626531.106007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10865 00000100:00080000:1.0:1778626531.106011:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626531.106014:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626531.106017:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626531.106018:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626531/real 1778626531] req@ffff954350195500 x1865013828401152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626531.106027:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828401664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626531.106038:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828401152/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626531.106074:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626531.106078:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626531.106082:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828401152:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626531.106089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626531.106107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626531.106115:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626531/real 1778626531] req@ffff954350194a80 x1865013828401280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626531.106124:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828401280/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626531.106140:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626531.106142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626531.106144:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626531.106163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401408:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626531.106177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626531.106183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626531/real 1778626531] req@ffff954350194700 x1865013828401408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626531.106191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828401408/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626531.106196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626531.106198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626531.106200:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401408:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626531.106204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626531.106213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626531.106218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626531/real 1778626531] req@ffff954350197480 x1865013828401536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626531.106225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828401536/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626531.106230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626531.106231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626531.106238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626531.106242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401664:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626531.106254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626531.106260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626531/real 1778626531] req@ffff954350196d80 x1865013828401664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626531.106267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828401664/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626531.106273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626531.106276:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626531.106278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401664:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626531.297812:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626532.065826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626532.321850:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626533.089754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626533.345774:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626534.113789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626534.369812:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626535.137748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626535.393755:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626536.161752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626536.225807:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626536.225825:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626536.225829:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626536.225838:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10870 00000100:00080000:1.0:1778626536.225848:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626536.225853:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626536.225897:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828401792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626536.225942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626536.225946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626536.225948:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626536.225952:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10870 00000100:00080000:1.0:1778626536.225969:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626536.225974:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626536.225984:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626536.225997:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828401792:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626536.226009:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828401920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626536.226024:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626536.226028:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626536.226030:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626536.226033:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10870 00000100:00080000:1.0:1778626536.226037:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626536.226040:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626536.226045:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626536.226063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828402048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626536.226070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626536.226078:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626536.226081:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626536.226084:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626536.226089:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10870 00000100:00080000:1.0:1778626536.226098:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626536.226104:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626536.226109:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626536.226113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626536/real 1778626536] req@ffff954350197100 x1865013828401792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626536.226122:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828402176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626536.226131:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626536.226135:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778626536.226135:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828401792/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626536.226139:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626536.226146:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10870 00000100:00080000:1.0:1778626536.226161:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626536.226164:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778626536.226164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778626536.226171:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626536.226172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626536.226178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828401792:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626536.226182:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828402304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626536.226199:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626536.226219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626536.226237:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626536/real 1778626536] req@ffff954350194000 x1865013828401920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626536.226246:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828401920/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626536.226251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626536.226254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626536.226256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828401920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626536.226260:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402048:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626536.226272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626536.226277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626536/real 1778626536] req@ffff954350196d80 x1865013828402048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626536.226285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828402048/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626536.226290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626536.226291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626536.226293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402048:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626536.226297:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626536.226307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626536.226312:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626536/real 1778626536] req@ffff954350197480 x1865013828402176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626536.226319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828402176/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626536.226328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626536.226330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626536.226332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626536.226336:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402304:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626536.226350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626536.226360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626536/real 1778626536] req@ffff954350194700 x1865013828402304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626536.226373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828402304/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626536.226382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626536.226385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626536.226388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402304:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626536.417771:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626537.185726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626537.441882:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626538.209745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626538.466128:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626539.233743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626539.490024:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626540.257756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626540.513963:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626541.281794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626541.345817:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626541.345828:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626541.345833:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626541.345843:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10875 00000100:00080000:2.0:1778626541.345872:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626541.345880:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626541.345923:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788000 x1865013828402432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626541.345953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626541.345972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626541.345974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626541.345977:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10875 00000100:00080000:2.0:1778626541.345981:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626541.345984:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626541.345993:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626541.346004:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828402560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626541.346011:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626541.346014:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626541.346015:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626541.346018:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10875 00000100:00080000:2.0:1778626541.346020:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626541.346023:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626541.346025:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626541.346030:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828402688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626541.346036:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626541.346039:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626541.346040:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626541.346044:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10875 00000100:00080000:2.0:1778626541.346054:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626541.346057:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626541.346066:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828402432:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778626541.346067:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626541.346079:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828402816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626541.346087:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626541.346090:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626541.346091:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626541.346094:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10875 00000100:00080000:2.0:1778626541.346097:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626541.346100:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626541.346103:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626541.346112:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828402944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626541.346164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626541.346208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626541/real 1778626541] req@ffff954358788000 x1865013828402432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626541.346231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828402432/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626541.346244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626541.346249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626541.346255:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828402432:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626541.346264:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626541.346290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626541.346304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626541/real 1778626541] req@ffff954358789180 x1865013828402560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626541.346318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828402560/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626541.346328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626541.346332:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626541.346335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626541.346342:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402688:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626541.346365:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626541.346389:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626541/real 1778626541] req@ffff954358789c00 x1865013828402688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626541.346414:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828402688/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626541.346424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626541.346428:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626541.346431:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402688:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626541.346468:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626541.346490:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626541.346501:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626541/real 1778626541] req@ffff95435878b800 x1865013828402816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626541.346513:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828402816/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626541.346522:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626541.346526:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626541.346530:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626541.346536:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402944:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626541.346558:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626541.346569:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626541/real 1778626541] req@ffff95435878a300 x1865013828402944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626541.346581:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828402944/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626541.346596:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626541.346600:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626541.346603:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828402944:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626541.538017:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626542.305744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626542.562061:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626543.329778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626543.585782:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626544.353754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626544.609826:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626545.377768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626545.633873:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626546.401739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626546.465813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626546.465820:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626546.465824:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626546.465831:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10880 00000100:00080000:0.0:1778626546.465841:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626546.465846:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626546.465888:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828403072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626546.465911:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626546.465914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626546.465917:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626546.465921:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10880 00000100:00080000:0.0:1778626546.465925:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626546.465929:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626546.465938:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626546.465947:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828403072:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626546.465949:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828403200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626546.465974:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626546.465977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626546.465979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626546.465982:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10880 00000100:00080000:0.0:1778626546.465985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626546.465988:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626546.465993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626546.466013:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828403328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626546.466016:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626546.466022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626546.466024:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626546.466026:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626546.466029:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10880 00000100:00080000:0.0:1778626546.466034:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626546.466038:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626546.466042:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626546.466049:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828403456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626546.466050:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626546/real 1778626546] req@ffff954342f48e00 x1865013828403072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626546.466059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626546.466062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626546.466063:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626546.466077:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10880 00000100:00100000:2.0:1778626546.466079:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828403072/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626546.466086:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626546.466089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626546.466090:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626546.466094:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626546.466094:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626546.466098:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828403072:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626546.466104:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828403584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626546.466122:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626546.466151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626546.466162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626546/real 1778626546] req@ffff954342f4aa00 x1865013828403200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626546.466181:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828403200/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626546.466186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626546.466188:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626546.466190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626546.466194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403328:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626546.466206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626546.466212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626546/real 1778626546] req@ffff954342f48380 x1865013828403328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626546.466219:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828403328/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626546.466224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626546.466226:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626546.466227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403328:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626546.466231:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626546.466250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626546.466255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626546/real 1778626546] req@ffff954342f48000 x1865013828403456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626546.466272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828403456/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626546.466278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626546.466280:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626546.466281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626546.466285:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403584:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626546.466296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626546.466301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626546/real 1778626546] req@ffff954342f49180 x1865013828403584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626546.466308:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828403584/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626546.466313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626546.466315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626546.466317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403584:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626546.657844:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626547.425795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626547.681836:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626548.449774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626548.705835:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626549.473869:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626549.729823:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626550.497783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626550.753800:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626551.521774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626551.585778:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626551.585786:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626551.585790:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626551.585798:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10886 00000100:00080000:1.0:1778626551.585820:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626551.585826:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626551.585855:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828403712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626551.585883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626551.585888:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626551.585890:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626551.585894:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10886 00000100:00080000:1.0:1778626551.585899:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626551.585904:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626551.585914:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626551.585923:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828403712:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626551.585929:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828403840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626551.585939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626551.585941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626551.585943:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626551.585946:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10886 00000100:00080000:1.0:1778626551.585960:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626551.585962:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626551.585975:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626551.585984:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828403968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626551.585991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626551.585998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626551.586000:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626551.586003:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10886 00000100:00080000:1.0:1778626551.586007:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778626551.586011:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626551.586014:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626551.586020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626551.586029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828404096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626551.586036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626551.586038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626551.586040:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626551.586044:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10886 00000100:00080000:1.0:1778626551.586048:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626551.586051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626551/real 1778626551] req@ffff954350194a80 x1865013828403712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626551.586055:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626551.586067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626551.586087:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828403712/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626551.586095:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828404224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626551.586120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626551.586125:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626551.586129:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828403712:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626551.586149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626551.586170:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626551.586183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626551/real 1778626551] req@ffff954350195500 x1865013828403840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626551.586196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828403840/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626551.586213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626551.586216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626551.586220:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626551.586226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403968:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626551.586243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626551.586251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626551/real 1778626551] req@ffff954350195c00 x1865013828403968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626551.586259:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828403968/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626551.586264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626551.586266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626551.586269:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828403968:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626551.586284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626551.586308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626551.586319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626551/real 1778626551] req@ffff954350197800 x1865013828404096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626551.586332:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828404096/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626551.586340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626551.586343:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626551.586347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626551.586353:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404224:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626551.586368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626551.586379:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626551/real 1778626551] req@ffff954350194380 x1865013828404224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626551.586386:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828404224/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626551.586392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626551.586394:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626551.586396:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404224:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626551.777837:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626552.545744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626552.801880:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626553.569800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626553.825836:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626554.593774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626554.849833:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626555.617828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626555.873888:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626556.641776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626556.705788:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626556.705800:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626556.705805:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626556.705839:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10891 00000100:00080000:2.0:1778626556.705852:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626556.705859:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626556.705899:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828404352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626556.705935:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626556.705940:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626556.705942:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626556.705947:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10891 00000100:00080000:2.0:1778626556.705952:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626556.705956:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626556.705969:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626556.705985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828404480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626556.705992:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828404352:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626556.706004:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626556.706008:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626556.706009:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626556.706013:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10891 00000100:00080000:2.0:1778626556.706016:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626556.706020:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626556.706024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626556.706035:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828404608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626556.706053:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626556.706056:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626556.706057:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626556.706061:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10891 00000100:00080000:2.0:1778626556.706064:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626556.706067:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626556.706071:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626556.706075:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626556.706085:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828404736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626556.706092:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626556.706095:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626556.706096:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626556.706099:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10891 00000100:00080000:2.0:1778626556.706101:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626556.706104:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626556.706107:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626556.706114:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828404864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626556.706126:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626556/real 1778626556] req@ffff954358789f80 x1865013828404352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626556.706149:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828404352/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626556.706163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626556.706167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626556.706172:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828404352:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626556.706204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626556.706238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626556.706251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626556/real 1778626556] req@ffff95435878a680 x1865013828404480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626556.706265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828404480/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626556.706275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626556.706279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626556.706282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626556.706288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404608:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626556.706307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626556.706318:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626556/real 1778626556] req@ffff95435878ad80 x1865013828404608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626556.706330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828404608/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626556.706338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626556.706342:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626556.706345:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404608:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626556.706352:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626556.706374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626556.706384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626556/real 1778626556] req@ffff95435878a300 x1865013828404736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626556.706411:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828404736/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626556.706419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626556.706422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626556.706424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626556.706428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404864:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626556.706443:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626556.706450:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626556/real 1778626556] req@ffff95435878b800 x1865013828404864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626556.706457:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828404864/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626556.706463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626556.706464:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626556.706466:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828404864:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778626556.897851:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626557.665763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626557.921843:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626558.689749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626558.945811:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626559.713807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626559.969831:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626560.737780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626560.993800:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626561.761782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626561.825874:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626561.825890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626561.825896:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626561.825910:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10896 00000100:00080000:2.0:1778626561.825951:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626561.825958:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626561.826002:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828404992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626561.826039:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626561.826044:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626561.826047:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626561.826052:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10896 00000100:00080000:2.0:1778626561.826057:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626561.826061:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626561.826072:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626561.826084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828405120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626561.826093:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626561.826097:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626561.826098:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626561.826101:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10896 00000100:00080000:2.0:1778626561.826104:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626561.826106:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626561.826110:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626561.826117:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788000 x1865013828405248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626561.826124:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626561.826133:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626561.826135:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626561.826139:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10896 00000100:00080000:2.0:1778626561.826143:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626561.826148:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626561.826156:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626561.826158:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828404992:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626561.826167:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828405376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626561.826177:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626561.826181:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626561.826183:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626561.826189:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10896 00000100:00080000:2.0:1778626561.826194:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626561.826200:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626561.826208:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626561.826224:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828405504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626561.826237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626561.826275:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626561/real 1778626561] req@ffff954358789c00 x1865013828404992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626561.826304:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828404992/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626561.826313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626561.826324:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626561.826328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828404992:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626561.826334:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626561.826351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626561.826359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626561/real 1778626561] req@ffff954358789180 x1865013828405120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626561.826367:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828405120/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626561.826372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626561.826375:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626561.826377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626561.826380:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405248:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626561.826390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626561.826395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626561/real 1778626561] req@ffff954358788000 x1865013828405248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626561.826402:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788000 x1865013828405248/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626561.826415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626561.826418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626561.826420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405248:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626561.826442:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626561.826470:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626561.826477:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626561/real 1778626561] req@ffff954358788e00 x1865013828405376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626561.826485:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828405376/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626561.826490:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626561.826503:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626561.826504:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626561.826508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405504:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626561.826517:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626561.826522:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626561/real 1778626561] req@ffff954358789500 x1865013828405504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626561.826529:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828405504/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626561.826534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626561.826536:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626561.826542:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405504:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626562.017850:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626562.785776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626563.041823:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626563.809778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626564.065804:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626564.833784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626565.089844:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626565.857799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626566.113859:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626566.881742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626566.945798:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626566.945809:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626566.945813:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626566.945823:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10901 00000100:00080000:2.0:1778626566.945838:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626566.945845:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626566.945895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828405632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626566.945941:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626566.945945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626566.945946:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626566.945950:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10901 00000100:00080000:2.0:1778626566.945953:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626566.945956:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626566.945965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626566.945978:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828405760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626566.945987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626566.945990:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626566.945992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626566.945996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10901 00000100:00080000:2.0:1778626566.946001:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626566.946004:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828405632:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626566.946007:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626566.946019:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626566.946035:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828405888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626566.946043:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626566.946046:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626566.946047:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626566.946051:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10901 00000100:00080000:2.0:1778626566.946059:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626566.946063:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626566.946068:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626566.946077:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828406016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626566.946085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626566.946098:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626566.946101:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626566.946105:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626566.946109:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10901 00000100:00080000:2.0:1778626566.946112:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626566.946117:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626566.946122:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626566.946133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626566/real 1778626566] req@ffff954358788a80 x1865013828405632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626566.946134:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828406144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626566.946222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828405632/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626566.946232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626566.946235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626566.946238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828405632:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626566.946258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626566.946274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626566.946282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626566/real 1778626566] req@ffff954358789880 x1865013828405760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626566.946296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828405760/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626566.946302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626566.946305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626566.946321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626566.946326:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405888:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626566.946340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626566.946347:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626566/real 1778626566] req@ffff954358788700 x1865013828405888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626566.946369:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828405888/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626566.946375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626566.946377:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626566.946379:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828405888:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626566.946383:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626566.946397:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626566.946404:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626566/real 1778626566] req@ffff95435878bb80 x1865013828406016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626566.946412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828406016/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626566.946416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626566.946418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626566.946420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626566.946424:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406144:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626566.946434:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626566.946458:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626566/real 1778626566] req@ffff954358788380 x1865013828406144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626566.946466:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828406144/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626566.946471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626566.946473:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626566.946475:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406144:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626567.137793:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626567.905761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626568.161818:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626568.929823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626569.185905:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626569.953779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626570.209837:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626570.977823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626571.233932:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626572.001732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626572.065746:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626572.065753:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626572.065757:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626572.065765:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10906 00000100:00080000:2.0:1778626572.065774:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626572.065779:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626572.065812:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828406272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626572.065836:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626572.065863:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626572.065865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626572.065868:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10906 00000100:00080000:2.0:1778626572.065872:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626572.065876:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626572.065884:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626572.065896:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828406272:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626572.065896:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828406400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626572.065921:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626572.065924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626572.065925:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626572.065928:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10906 00000100:00080000:2.0:1778626572.065930:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626572.065932:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626572.065935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626572.065943:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828406528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626572.065949:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626572.065952:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626572.065953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626572.065955:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10906 00000100:00080000:2.0:1778626572.065957:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626572.065960:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626572.065967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626572.065976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828406656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626572.065983:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626572.065985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626572.065986:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626572.065989:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10906 00000100:00080000:2.0:1778626572.065992:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626572.065995:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626572.065999:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626572.066002:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626572.066008:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828406784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626572.066048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406528:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626572.066063:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626572.066079:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406784:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626572.257814:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626573.025741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626573.281793:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626574.049737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626574.305828:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626575.073760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778626575.138094:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778626575.138205:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778626575.138249:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626575.138306:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626575.138314:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626575.138321:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626575.138328:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626575.138394:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626572/real 1778626575] req@ffff95435878bb80 x1865013828406784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626575.138434:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828406784/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626575.138479:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626575.138488:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626575.138495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406784:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626575.138507:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626572/real 1778626575] req@ffff95435878b480 x1865013828406400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626575.138528:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828406400/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626575.138540:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626575.138546:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626575.138551:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626575.138561:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626572/real 1778626575] req@ffff95435878b100 x1865013828406528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626575.138577:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828406528/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626575.138588:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626575.138593:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626575.138598:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406528:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626575.138618:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626572/real 1778626575] req@ffff954358788380 x1865013828406656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626575.138634:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828406656/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626575.138654:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626575.138659:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626575.138664:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828406656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626575.138709:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626572/real 1778626575] req@ffff95435878aa00 x1865013828406272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626575.138729:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828406272/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626575.138739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626575.138743:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626575.138748:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828406272:192.168.204.102@tcp:250:kworker.0 00010000:00010000:1.0:1778626575.329841:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626576.097757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626576.353815:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626577.121779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626577.185766:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626577.185775:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626577.185779:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626577.185786:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10911 00000100:00080000:2.0:1778626577.185796:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626577.185801:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626577.185830:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828406912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626577.185853:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626577.185856:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626577.185857:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626577.185860:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10911 00000100:00080000:2.0:1778626577.185876:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626577.185879:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626577.185886:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828406912:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778626577.185890:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626577.185906:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828407040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626577.185918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626577.185921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626577.185923:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626577.185926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10911 00000100:00080000:2.0:1778626577.185930:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626577.185933:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626577.185938:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778626577.185954:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626577.185960:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828407168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626577.185971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626577.185974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626577.185977:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626577.185981:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10911 00000100:00080000:2.0:1778626577.185985:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626577.185989:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626577.185995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626577.186001:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626577/real 1778626577] req@ffff954358788700 x1865013828406912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626577.186009:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828407296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626577.186033:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828406912/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626577.186050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626577.186054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626577.186055:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626577.186058:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626577.186062:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778626577.186067:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10911 00000100:00100000:0.0:1778626577.186070:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828406912:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626577.186074:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626577.186080:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626577.186084:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626577.186106:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828407424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626577.186110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626577.186145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626577.186178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626577/real 1778626577] req@ffff954358789880 x1865013828407040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626577.186195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828407040/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626577.186205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626577.186208:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626577.186211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626577.186217:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407168:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626577.186239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626577.186252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626577/real 1778626577] req@ffff954358788a80 x1865013828407168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626577.186264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828407168/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626577.186271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626577.186275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626577.186278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407168:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626577.186285:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626577.186309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626577.186326:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626577/real 1778626577] req@ffff954358789500 x1865013828407296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626577.186347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828407296/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626577.186355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626577.186359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626577.186362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626577.186392:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407424:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626577.186421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626577.186432:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626577/real 1778626577] req@ffff954358788e00 x1865013828407424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626577.186445:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828407424/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626577.186453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626577.186457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626577.186466:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407424:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626577.377803:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626578.145788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626578.401847:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626579.169748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626579.425827:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626580.193774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626580.449807:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626581.217758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626581.473824:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626582.241782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626582.305776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626582.305785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626582.305789:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626582.305797:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10916 00000100:00080000:1.0:1778626582.305807:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626582.305812:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626582.305844:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828407552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626582.305868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626582.305871:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626582.305872:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626582.305875:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10916 00000100:00080000:1.0:1778626582.305893:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626582.305897:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626582.305905:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626582.305915:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196680 x1865013828407680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626582.305920:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828407552:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626582.305926:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626582.305929:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626582.305930:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626582.305934:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10916 00000100:00080000:1.0:1778626582.305936:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626582.305938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626582.305941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626582.305949:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828407808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626582.305956:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626582.305958:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626582.305959:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626582.305961:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10916 00000100:00080000:1.0:1778626582.305971:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626582.305974:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626582.305978:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626582.305978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626582.305989:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828407936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626582.306005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626582.306007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626582.306009:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626582.306013:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626582/real 1778626582] req@ffff954350196a00 x1865013828407552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626582.306014:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10916 00000100:00080000:1.0:1778626582.306017:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626582.306020:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626582.306024:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626582.306036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828408064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626582.306066:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828407552/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626582.306080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626582.306085:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626582.306089:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828407552:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626582.306110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626582.306129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626582.306138:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626582/real 1778626582] req@ffff954350196680 x1865013828407680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626582.306147:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196680 x1865013828407680/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626582.306153:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626582.306155:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626582.306157:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626582.306161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407808:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626582.306171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626582.306176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626582/real 1778626582] req@ffff954350194e00 x1865013828407808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626582.306184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828407808/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626582.306198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626582.306200:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626582.306202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407808:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626582.306206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626582.306219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626582.306225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626582/real 1778626582] req@ffff954350195880 x1865013828407936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626582.306232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828407936/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626582.306237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626582.306239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626582.306240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828407936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626582.306244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408064:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626582.306254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626582.306259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626582/real 1778626582] req@ffff954350197b80 x1865013828408064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626582.306266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828408064/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626582.306270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626582.306272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626582.306278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408064:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626582.497822:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626583.265760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626583.521839:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626584.289766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626584.545806:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626585.313767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626585.569862:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626586.337788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626586.594005:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626587.361799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626587.425828:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626587.425839:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626587.425844:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626587.425855:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10921 00000100:00080000:1.0:1778626587.425869:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626587.425876:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626587.425923:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828408192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626587.425962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626587.425966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626587.425986:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626587.425990:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10921 00000100:00080000:1.0:1778626587.425995:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626587.425998:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626587.426008:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626587.426020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828408320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626587.426028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626587.426031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626587.426032:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626587.426036:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10921 00000100:00080000:1.0:1778626587.426038:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626587.426040:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626587.426045:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626587.426054:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828408192:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626587.426058:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828408448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626587.426075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626587.426078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626587.426079:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626587.426083:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10921 00000100:00080000:1.0:1778626587.426086:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626587.426088:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626587.426092:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626587.426109:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828408576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626587.426115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626587.426118:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626587.426119:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626587.426121:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10921 00000100:00080000:1.0:1778626587.426127:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626587.426129:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626587.426133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626587.426140:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626587.426149:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828408704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626587.426182:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626587/real 1778626587] req@ffff954350195f80 x1865013828408192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626587.426198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828408192/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626587.426207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626587.426211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626587.426214:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828408192:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626587.426219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626587.426234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626587.426241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626587/real 1778626587] req@ffff954350195180 x1865013828408320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626587.426249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828408320/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626587.426254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626587.426256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626587.426271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626587.426292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408448:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626587.426305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626587.426312:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626587/real 1778626587] req@ffff954350197b80 x1865013828408448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626587.426319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828408448/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626587.426324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626587.426326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626587.426328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408448:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626587.426332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626587.426342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626587.426347:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626587/real 1778626587] req@ffff954350195880 x1865013828408576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626587.426354:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828408576/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626587.426359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626587.426361:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626587.426362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626587.426366:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408704:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626587.426374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626587.426387:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626587/real 1778626587] req@ffff954350194e00 x1865013828408704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626587.426394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828408704/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626587.426400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626587.426402:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626587.426404:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828408704:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626587.617855:0:157327:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626588.385773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626588.641849:0:157326:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626589.409733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626589.665840:0:189176:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 48/48 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1778626589.665885:0:157327:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 48 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1778626589.665894:0:157327:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9543436d7000 lock: 00000000273a5d3a/0x56379fd28fe4ee83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x19d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0xa0dae0fc83604834 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1778626589.665930:0:189176:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 48/48 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1778626589.665970:0:157326:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 48 locks of 0 total in blp. (0 blwis in pool) 00000008:00010000:1.0:1778626589.665972:0:157327:0:(osc_cache.c:2021:try_to_add_extent_for_io()) ### extent: 000000009e8cc664 ns: lustre-OST0000-osc-ffff9543436d7000 lock: 00000000273a5d3a/0x56379fd28fe4ee83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x19d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604834 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:1.0:1778626589.665985:0:157327:0:(osc_cache.c:1176:osc_extent_make_ready()) ### extent: 000000009e8cc664 ns: lustre-OST0000-osc-ffff9543436d7000 lock: 00000000273a5d3a/0x56379fd28fe4ee83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x19d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604834 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1778626589.666032:0:157327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9543436d7000 lock: 00000000273a5d3a/0x56379fd28fe4ee83 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x19d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604834 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:1.0:1778626589.666129:0:157327:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff954350196680 - object 0x280000401:6611 offset 0<>7 00000100:00100000:1.0:1778626589.666176:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:1.0:1778626589.666193:0:2404:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff954350196680 x1865013828408832/t0(0) o4->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:6/4 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'grep.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1778626589.666247:0:189194:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626589.666249:0:189194:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626589.666250:0:189194:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1778626589.666269:0:157326:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9543436d7000 lock: 000000002decd78f/0x56379fd28fe4eea6 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1994:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0xa0dae0fc83604896 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1778626589.666314:0:157326:0:(osc_cache.c:2021:try_to_add_extent_for_io()) ### extent: 00000000a9814215 ns: lustre-OST0001-osc-ffff9543436d7000 lock: 000000002decd78f/0x56379fd28fe4eea6 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1994:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604896 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00010000:3.0:1778626589.666322:0:157326:0:(osc_cache.c:1176:osc_extent_make_ready()) ### extent: 00000000a9814215 ns: lustre-OST0001-osc-ffff9543436d7000 lock: 000000002decd78f/0x56379fd28fe4eea6 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1994:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604896 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1778626589.666343:0:157326:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9543436d7000 lock: 000000002decd78f/0x56379fd28fe4eea6 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x1994:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604896 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:3.0:1778626589.666376:0:157326:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff95434701fb80 - object 0x2c0000401:6548 offset 0<>8 00000100:00100000:3.0:1778626589.666410:0:2406:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1778626589.666419:0:2406:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff95434701fb80 x1865013828408960/t0(0) o4->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:6/4 lens 496/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'grep.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1778626590.433780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778626590.689823:0:189194:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626591.457760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626591.713920:0:189176:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 202/202 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1778626591.713961:0:189194:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 202 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1778626591.714250:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1778626591.714252:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1778626591.714254:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1778626591.714275:0:189194:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9543436d7000 lock: ffff954348f5a000/0x56379fd28fe4eeb4 lrc: 2/0,0 mode: CR/CR res: [0x20002d2c1:0x64:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xa0dae0fc836048b2 expref: -99 pid: 188454 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00100000:2.0:1778626591.714340:0:189194:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff954345753b80 x1865013828339712/t395136991537(395136991537) o101->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1778626003 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'grep.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1778626591.714349:0:189194:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9543436d7000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1778626591.714353:0:189194:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff954358788000 x1865013828409216/t0(0) o35->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00080000:2.0:1778626591.714369:0:189194:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff954358788000 x1865013828409216/t0(0) o35->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1778626591.714387:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626592.481785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626592.545836:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626592.545862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626592.545867:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626592.545879:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10927 00000100:00080000:1.0:1778626592.545910:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626592.545917:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626592.545963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828409344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626592.545994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626592.545997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626592.545998:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626592.546002:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10927 00000100:00080000:1.0:1778626592.546004:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626592.546006:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626592.546015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626592.546025:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828409472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626592.546033:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626592.546053:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828409344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626592.546062:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626592.546064:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626592.546068:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10927 00000100:00080000:1.0:1778626592.546071:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626592.546075:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626592.546080:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626592.546089:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828409600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626592.546097:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626592.546100:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626592.546101:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626592.546105:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10927 00000100:00080000:1.0:1778626592.546117:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626592.546119:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626592.546122:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626592.546131:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828409728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626592.546137:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626592.546142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626592.546142:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626592.546147:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626592.546150:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10927 00000100:00080000:1.0:1778626592.546152:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626592.546155:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626592.546167:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626592.546180:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828409856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626592.546187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626592/real 1778626592] req@ffff954350196a00 x1865013828409344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626592.546211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828409344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626592.546220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626592.546231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626592.546234:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828409344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626592.546241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626592.546266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626592.546275:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626592/real 1778626592] req@ffff954350194380 x1865013828409472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626592.546284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828409472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626592.546290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626592.546293:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626592.546295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626592.546320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626592.546331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626592.546337:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626592/real 1778626592] req@ffff954350197800 x1865013828409600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626592.546344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828409600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626592.546349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626592.546351:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626592.546353:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626592.546357:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626592.546376:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626592.546382:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626592/real 1778626592] req@ffff954350195c00 x1865013828409728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626592.546390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828409728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626592.546395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626592.546396:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626592.546398:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626592.546402:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626592.546411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626592.546417:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626592/real 1778626592] req@ffff954350195500 x1865013828409856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626592.546424:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828409856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626592.546430:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626592.546432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626592.546433:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828409856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626592.737806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626592.737856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626593.505773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626593.761760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626593.761800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626594.529832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626594.785812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626594.785845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626595.553750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626595.809746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626595.809773:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626596.577753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626596.833774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626596.833811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626597.601769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626597.665736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626597.665759:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626597.665763:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626597.665771:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10932 00000100:00080000:1.0:1778626597.665781:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626597.665798:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626597.665832:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828409984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626597.665868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626597.665871:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626597.665872:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626597.665875:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10932 00000100:00080000:1.0:1778626597.665878:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626597.665880:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626597.665888:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626597.665898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828410112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626597.665906:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828409984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626597.665908:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626597.665926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626597.665927:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626597.665930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10932 00000100:00080000:1.0:1778626597.665947:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626597.665951:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626597.665955:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626597.665964:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828410240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626597.665972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626597.665974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626597.665975:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626597.665977:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10932 00000100:00080000:1.0:1778626597.665980:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626597.665984:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778626597.665987:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626597.665990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626597.665998:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828410368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626597.666005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626597.666008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626597.666009:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626597.666012:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10932 00000100:00080000:1.0:1778626597.666014:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626597.666016:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626597.666020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626597.666027:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828410496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626597.666032:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626597/real 1778626597] req@ffff954350194a80 x1865013828409984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626597.666058:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828409984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626597.666071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626597.666076:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626597.666081:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828409984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626597.666105:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626597.666128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626597.666137:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626597/real 1778626597] req@ffff954350194700 x1865013828410112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626597.666146:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828410112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626597.666152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626597.666154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626597.666163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626597.666168:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626597.666182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626597.666188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626597/real 1778626597] req@ffff954350197480 x1865013828410240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626597.666196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828410240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626597.666201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626597.666203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626597.666205:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626597.666209:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626597.666218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626597.666223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626597/real 1778626597] req@ffff954350196d80 x1865013828410368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626597.666230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828410368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626597.666236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626597.666237:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626597.666239:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626597.666242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626597.666251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626597.666262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626597/real 1778626597] req@ffff954350194000 x1865013828410496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626597.666269:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828410496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626597.666275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626597.666277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626597.666279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626597.857759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626597.857821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626598.625823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626598.881762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626598.881764:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626599.649797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626599.905741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626599.905791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626600.673760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626600.929771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626600.929794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626601.697771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626601.953758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626601.953802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626602.721744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626602.785775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626602.785783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626602.785787:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626602.785795:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10937 00000100:00080000:1.0:1778626602.785805:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626602.785811:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626602.785843:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828410624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626602.785867:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626602.785871:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626602.785873:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626602.785877:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10937 00000100:00080000:1.0:1778626602.785881:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626602.785885:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626602.785896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626602.785910:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828410624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626602.785915:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828410752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626602.785929:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626602.785932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626602.785934:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626602.785938:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10937 00000100:00080000:1.0:1778626602.785942:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626602.785946:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778626602.785965:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626602.785970:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626602.785986:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828410880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626602.785996:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626602/real 1778626602] req@ffff954350197100 x1865013828410624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626602.786000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626602.786009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626602.786012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626602.786018:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10937 00000100:00080000:1.0:1778626602.786023:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626602.786028:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828410624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626602.786031:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626602.786040:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:2.0:1778626602.786044:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626602.786050:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778626602.786056:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828411008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626602.786056:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828410624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626602.786072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626602.786076:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626602.786078:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626602.786082:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10937 00000100:00080000:1.0:1778626602.786086:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626602.786090:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626602.786095:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626602.786095:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778626602.786113:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828411136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626602.786129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626602.786145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626602/real 1778626602] req@ffff954350194000 x1865013828410752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626602.786167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828410752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626602.786176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626602.786179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626602.786183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626602.786189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626602.786211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626602.786263:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626602/real 1778626602] req@ffff954350196d80 x1865013828410880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626602.786286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828410880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626602.786292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626602.786294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626602.786296:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828410880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626602.786301:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626602.786329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626602.786348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626602/real 1778626602] req@ffff954350197480 x1865013828411008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626602.786372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828411008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626602.786399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626602.786402:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626602.786437:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626602.786491:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626602.786515:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626602.786535:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626602/real 1778626602] req@ffff954350194700 x1865013828411136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626602.786543:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828411136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626602.786561:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626602.786565:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626602.786568:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626602.977744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626602.977765:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626603.745749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626604.001898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626604.002011:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626604.769750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626605.025932:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626605.025988:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626605.793789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626606.050091:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626606.050253:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626606.817739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778626607.074246:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626607.074609:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626607.841802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626607.905754:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626607.905768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626607.905796:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626607.905810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10942 00000100:00080000:1.0:1778626607.905826:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626607.905835:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626607.905879:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828411264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626607.905908:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626607.905912:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626607.905915:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626607.905919:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10942 00000100:00080000:1.0:1778626607.905923:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626607.905928:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626607.905941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626607.905958:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828411392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626607.905958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828411264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626607.905975:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626607.905980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626607.905982:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626607.905986:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10942 00000100:00080000:1.0:1778626607.905989:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626607.905994:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626607.905999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626607.906012:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828411520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626607.906035:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626607.906039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626607.906041:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626607.906046:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10942 00000800:00000100:2.0:1778626607.906051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626607.906054:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626607.906061:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626607.906068:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626607.906081:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828411648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626607.906091:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626607/real 1778626607] req@ffff954350194a80 x1865013828411264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626607.906094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626607.906100:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626607.906102:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626607.906109:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10942 00000100:00080000:1.0:1778626607.906115:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626607.906119:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626607.906125:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626607.906128:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828411264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626607.906139:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828411776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626607.906164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626607.906171:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626607.906176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828411264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626607.906211:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626607.906239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626607.906251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626607/real 1778626607] req@ffff954350195500 x1865013828411392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626607.906264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828411392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626607.906273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626607.906276:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626607.906279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626607.906286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626607.906307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626607.906318:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626607/real 1778626607] req@ffff954350195c00 x1865013828411520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626607.906331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828411520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626607.906337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626607.906339:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626607.906341:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626607.906352:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626607.906370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626607.906377:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626607/real 1778626607] req@ffff954350197800 x1865013828411648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626607.906385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828411648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626607.906390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626607.906393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626607.906395:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626607.906398:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626607.906408:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626607.906413:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626607/real 1778626607] req@ffff954350194380 x1865013828411776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626607.906420:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828411776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626607.906425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626607.906427:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626607.906429:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828411776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626608.097764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626608.097885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626608.865759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626609.121915:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626609.121985:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626609.889817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626610.145796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626610.145950:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626610.913762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626611.169746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626611.169784:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626611.937802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626612.193755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626612.193837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626612.961743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626613.025778:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626613.025788:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626613.025812:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626613.025822:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10947 00000100:00080000:2.0:1778626613.025833:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626613.025839:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626613.025867:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828411904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626613.025894:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626613.025897:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626613.025898:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626613.025901:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10947 00000100:00080000:2.0:1778626613.025904:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626613.025906:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626613.025913:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626613.025922:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828412032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626613.025930:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626613.025932:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626613.025933:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626613.025936:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10947 00000100:00080000:2.0:1778626613.025938:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626613.025940:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626613.025943:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626613.025946:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828411904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626613.025953:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828412160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626613.025970:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626613.025972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626613.025974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626613.025977:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10947 00000100:00080000:2.0:1778626613.025980:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626613.025982:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626613.025986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626613.025994:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828412288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626613.026000:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626613.026003:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626613.026004:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626613.026006:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10947 00000100:00080000:2.0:1778626613.026009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778626613.026010:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626613.026013:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626613.026016:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626613.026026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828412416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626613.026054:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626613/real 1778626613] req@ffff954358789180 x1865013828411904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626613.026069:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828411904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626613.026077:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626613.026080:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626613.026083:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828411904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626613.026088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626613.026104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626613.026112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626613/real 1778626613] req@ffff954358789c00 x1865013828412032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626613.026120:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828412032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626613.026125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626613.026128:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626613.026130:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626613.026146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626613.026156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626613.026162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626613/real 1778626613] req@ffff95435878b800 x1865013828412160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626613.026169:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828412160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626613.026183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626613.026186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626613.026188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626613.026192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626613.026205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626613.026211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626613/real 1778626613] req@ffff95435878a300 x1865013828412288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626613.026218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828412288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626613.026223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626613.026225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626613.026226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626613.026230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626613.026239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626613.026244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626613/real 1778626613] req@ffff95435878ad80 x1865013828412416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626613.026251:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828412416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626613.026257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626613.026258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626613.026264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626613.217744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626613.217821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626613.985790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626614.241759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626614.241845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626615.009734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626615.265740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626615.265834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626616.033780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626616.289757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626616.289821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626617.057775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626617.313757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626617.313842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626618.081821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626618.145781:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626618.145792:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626618.145796:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626618.145807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10952 00000100:00080000:2.0:1778626618.145820:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626618.145826:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626618.145869:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828412544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626618.145897:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626618.145900:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626618.145901:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626618.145905:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10952 00000100:00080000:2.0:1778626618.145907:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626618.145909:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626618.145918:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626618.145926:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828412672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626618.145933:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626618.145935:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626618.145937:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626618.145939:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10952 00000100:00080000:2.0:1778626618.145940:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626618.145943:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626618.145945:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626618.145972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828412800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626618.145980:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626618.145985:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828412544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626618.145985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626618.145986:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626618.145994:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10952 00000100:00080000:2.0:1778626618.145997:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626618.146000:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626618.146003:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626618.146013:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828412928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626618.146019:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626618.146022:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626618.146023:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626618.146025:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10952 00000100:00080000:2.0:1778626618.146027:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626618.146029:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626618.146033:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626618.146039:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828413056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626618.146063:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626618.146111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626618/real 1778626618] req@ffff95435878a680 x1865013828412544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626618.146129:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828412544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626618.146138:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626618.146141:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626618.146144:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828412544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626618.146149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626618.146165:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626618.146172:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626618/real 1778626618] req@ffff954358789f80 x1865013828412672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626618.146180:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828412672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626618.146185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626618.146187:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626618.146196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626618.146221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626618.146235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626618.146241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626618/real 1778626618] req@ffff95435878ad80 x1865013828412800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626618.146249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828412800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626618.146254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626618.146255:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626618.146257:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626618.146261:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626618.146271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626618.146277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626618/real 1778626618] req@ffff95435878a300 x1865013828412928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626618.146285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828412928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626618.146298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626618.146300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626618.146302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828412928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626618.146305:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626618.146316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626618.146328:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626618/real 1778626618] req@ffff95435878b800 x1865013828413056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626618.146336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828413056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626618.146356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626618.146359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626618.146361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413056:192.168.204.102@tcp:8:kworker.0 00010000:00010000:2.0:1778626618.337910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626618.337960:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626619.105821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626619.361755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626619.361854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626620.129744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626620.385764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626620.385815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626621.153807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626621.409743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626621.409853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626622.177767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778626622.433826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626622.433861:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626623.201858:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626623.265736:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626623.265755:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626623.265759:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626623.265767:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10957 00000100:00080000:2.0:1778626623.265778:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626623.265783:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626623.265816:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828413184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626623.265841:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626623.265844:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626623.265845:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626623.265848:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10957 00000100:00080000:2.0:1778626623.265851:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626623.265854:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626623.265862:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626623.265872:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828413312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626623.265887:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828413184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626623.265890:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626623.265904:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626623.265905:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626623.265909:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10957 00000100:00080000:2.0:1778626623.265911:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626623.265914:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626623.265930:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626623.265941:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828413440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626623.265949:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626623.265952:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626623.265953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778626623.265958:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626623.265958:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10957 00000100:00080000:2.0:1778626623.265964:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626623.265968:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626623.265972:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626623.265981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828413568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626623.265989:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626623.265994:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626623/real 1778626623] req@ffff954358789c00 x1865013828413184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626623.265994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626623.265995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626623.266013:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10957 00000100:00080000:2.0:1778626623.266018:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626623.266023:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626623.266029:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626623.266038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828413696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626623.266043:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828413184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626623.266061:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626623.266079:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626623.266084:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828413184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626623.266109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626623.266130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626623.266157:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626623/real 1778626623] req@ffff954358789180 x1865013828413312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626623.266168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828413312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626623.266174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626623.266177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626623.266180:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626623.266184:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626623.266198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626623.266215:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626623/real 1778626623] req@ffff954358788e00 x1865013828413440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626623.266223:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828413440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626623.266229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626623.266231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626623.266233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626623.266237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626623.266268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626623.266276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626623/real 1778626623] req@ffff954358789500 x1865013828413568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626623.266285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828413568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626623.266297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626623.266299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626623.266302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626623.266306:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626623.266321:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626623.266327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626623/real 1778626623] req@ffff954358788a80 x1865013828413696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626623.266336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828413696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626623.266342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626623.266344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626623.266346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626623.457759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626623.457829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626624.225767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626624.481757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626624.481857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626625.249862:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626625.505784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626625.505939:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626626.273862:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626626.529750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626626.529941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626627.297779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626627.553735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626627.553798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626628.321782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626628.385795:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626628.385804:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626628.385808:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626628.385817:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10962 00000100:00080000:3.0:1778626628.385828:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626628.385852:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626628.385885:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828413824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626628.385912:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626628.385915:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626628.385917:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626628.385920:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10962 00000100:00080000:3.0:1778626628.385924:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626628.385927:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626628.385934:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626628.385944:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828413952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626628.385953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626628.385956:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626628.385957:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626628.385960:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10962 00000100:00080000:3.0:1778626628.385963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626628.385965:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626628.385969:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626628.385973:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828413824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626628.385980:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828414080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626628.385989:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626628.385992:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626628.386000:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626628.386004:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10962 00000100:00080000:3.0:1778626628.386007:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626628.386010:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626628.386015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626628.386028:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828414208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626628.386035:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626628.386038:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626628.386040:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626628.386042:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10962 00000100:00080000:3.0:1778626628.386045:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626628.386047:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626628.386052:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778626628.386053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626628.386065:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828414336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626628.386094:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626628/real 1778626628] req@ffff95434701d180 x1865013828413824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626628.386118:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828413824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626628.386167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626628.386190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626628.386196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828413824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626628.386208:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626628.386233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626628.386263:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626628/real 1778626628] req@ffff95434701e680 x1865013828413952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626628.386276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828413952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626628.386284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626628.386287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626628.386290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828413952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626628.386312:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626628.386328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626628.386336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626628/real 1778626628] req@ffff95434701f480 x1865013828414080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626628.386347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828414080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626628.386354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626628.386357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626628.386359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626628.386373:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626628.386391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626628.386400:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626628/real 1778626628] req@ffff95434701ca80 x1865013828414208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626628.386425:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828414208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626628.386432:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626628.386434:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626628.386436:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626628.386441:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626628.386453:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626628.386460:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626628/real 1778626628] req@ffff95434701df80 x1865013828414336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626628.386468:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828414336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626628.386475:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626628.386477:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626628.386479:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626628.577737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626628.577849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626629.345807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626629.601817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626629.601901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626630.369771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626630.625749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626630.625874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626631.393766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626631.649786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626631.649895:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626632.417743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626632.673746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626632.673812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626633.441741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626633.505817:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626633.505825:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626633.505829:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626633.505837:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10968 00000100:00080000:2.0:1778626633.505846:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626633.505852:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626633.505885:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828414464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626633.505907:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626633.505910:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626633.505911:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626633.505914:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10968 00000100:00080000:2.0:1778626633.505916:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626633.505919:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626633.505926:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626633.505934:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828414592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626633.505942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626633.505959:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626633.505961:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626633.505962:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828414464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626633.505967:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10968 00000100:00080000:2.0:1778626633.505971:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626633.505974:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626633.505978:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626633.505989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828414720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626633.505998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626633.506000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626633.506001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626633.506004:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10968 00000100:00080000:2.0:1778626633.506006:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626633.506009:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626633.506012:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626633.506021:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626633.506022:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828414848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626633.506031:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626633.506033:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626633.506035:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626633.506065:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10968 00000100:00080000:2.0:1778626633.506069:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626633.506072:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626633.506081:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626633.506082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626633/real 1778626633] req@ffff954358789880 x1865013828414464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626633.506094:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828414976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626633.506102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828414464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626633.506112:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626633.506115:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626633.506119:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828414464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626633.506126:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626633.506144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626633.506153:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626633/real 1778626633] req@ffff954358788700 x1865013828414592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626633.506161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828414592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626633.506167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626633.506169:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626633.506171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626633.506191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626633.506202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626633.506208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626633/real 1778626633] req@ffff95435878aa00 x1865013828414720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626633.506223:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828414720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626633.506228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626633.506230:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626633.506232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626633.506236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626633.506249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626633.506255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626633/real 1778626633] req@ffff954358788380 x1865013828414848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626633.506262:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828414848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626633.506267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626633.506269:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626633.506271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626633.506274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626633.506284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626633.506289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626633/real 1778626633] req@ffff95435878b100 x1865013828414976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626633.506296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828414976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626633.506310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626633.506312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626633.506314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828414976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626633.697747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626633.697839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626634.465751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626634.721751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626634.721804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626635.489776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626635.745756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626635.745821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626636.513768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626636.769752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626636.769830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626637.537765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626637.793777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626637.793859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626638.561806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626638.625820:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626638.625833:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626638.625838:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626638.625849:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10973 00000100:00080000:2.0:1778626638.625861:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626638.625868:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626638.625922:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828415104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626638.625967:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626638.625971:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626638.625973:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626638.625977:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10973 00000100:00080000:2.0:1778626638.625980:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626638.625995:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626638.626003:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626638.626016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828415232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626638.626048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828415104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626638.626049:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626638.626057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626638.626062:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626638.626066:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10973 00000100:00080000:2.0:1778626638.626069:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626638.626072:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626638.626076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626638.626089:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828415360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626638.626097:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626638.626100:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626638.626101:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626638.626104:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10973 00000100:00080000:2.0:1778626638.626106:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626638.626109:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626638.626112:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626638.626121:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828415488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626638.626128:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626638.626131:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626638.626132:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626638.626137:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10973 00000100:00080000:2.0:1778626638.626149:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626638.626156:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626638.626163:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626638.626172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626638.626197:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828415616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626638.626239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626638.626270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626638.626301:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626638.817742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626638.817809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626639.585747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626639.841766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626639.841867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626640.609805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626640.865735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626640.865849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626641.633760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778626641.697920:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778626641.697956:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778626641.697969:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626641.698014:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626641.698055:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626641.698070:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626641.698074:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626638/real 1778626641] req@ffff95435878bb80 x1865013828415232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1778626641.698083:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626641.698109:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828415232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626641.698121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626641.698127:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626641.698140:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626641.698148:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626638/real 1778626641] req@ffff95435878b100 x1865013828415360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626641.698161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828415360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626641.698169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626641.698172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626641.698176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626641.698183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626638/real 1778626641] req@ffff954358788380 x1865013828415488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626641.698194:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828415488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626641.698202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626641.698205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626641.698208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626641.698215:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626638/real 1778626641] req@ffff95435878b480 x1865013828415104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626641.698226:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828415104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626641.698233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626641.698236:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626641.698245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828415104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626641.698273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626638/real 1778626641] req@ffff95435878aa00 x1865013828415616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626641.698286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828415616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626641.698294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626641.698298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626641.698301:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626641.889789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626641.889885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626642.657785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626642.913785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626642.913847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626643.681755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626643.745799:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626643.745811:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626643.745816:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626643.745827:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10978 00000100:00080000:2.0:1778626643.745866:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626643.745873:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626643.745902:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828415744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626643.745925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626643.745928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626643.745929:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626643.745949:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10978 00000100:00080000:2.0:1778626643.745952:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626643.745955:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626643.745963:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626643.746014:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828415872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626643.746023:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828415744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626643.746031:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626643.746035:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626643.746037:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626643.746042:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10978 00000100:00080000:2.0:1778626643.746047:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626643.746052:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626643.746058:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626643.746073:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828416000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626643.746084:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626643.746088:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626643.746089:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778626643.746095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626643.746096:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10978 00000100:00080000:2.0:1778626643.746104:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626643.746109:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626643.746116:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626643.746129:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828416128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626643.746138:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626643/real 1778626643] req@ffff954358788700 x1865013828415744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626643.746139:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626643.746147:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626643.746150:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626643.746162:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10978 00000100:00080000:2.0:1778626643.746170:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626643.746177:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626643.746185:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626643.746198:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828416256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626643.746204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828415744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626643.746226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626643.746232:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626643.746237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828415744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626643.746262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626643.746290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626643.746305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626643/real 1778626643] req@ffff954358789880 x1865013828415872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626643.746328:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828415872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626643.746339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626643.746343:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626643.746346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828415872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626643.746353:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626643.746378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626643.746391:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626643/real 1778626643] req@ffff954358788a80 x1865013828416000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626643.746404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828416000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626643.746414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626643.746417:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626643.746421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626643.746428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626643.746452:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626643.746472:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626643/real 1778626643] req@ffff954358789500 x1865013828416128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626643.746485:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828416128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626643.746508:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626643.746513:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626643.746516:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626643.746522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626643.746546:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626643.746559:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626643/real 1778626643] req@ffff954358788e00 x1865013828416256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626643.746571:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828416256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626643.746580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626643.746583:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626643.746587:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626643.937785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626643.937870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626644.705840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626644.961787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626644.961856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626645.729787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626645.985739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626645.985826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626646.753761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626647.009786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626647.009869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626647.777834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626648.033729:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626648.033815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626648.801829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626648.865829:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626648.865838:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626648.865843:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626648.865852:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10983 00000100:00080000:1.0:1778626648.865863:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626648.865870:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626648.865931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828416384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626648.865955:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626648.865958:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626648.865960:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626648.865964:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10983 00000100:00080000:1.0:1778626648.865967:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626648.865969:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626648.865978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626648.866005:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828416512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626648.866015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626648.866019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626648.866021:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626648.866024:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10983 00000100:00100000:0.0:1778626648.866035:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828416384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626648.866048:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626648.866054:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626648.866062:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626648.866082:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828416640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626648.866121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626648.866126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1778626648.866131:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626648.866131:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626648.866143:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10983 00000100:00080000:1.0:1778626648.866150:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626648.866171:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626648.866177:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626648.866191:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828416768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626648.866198:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626648/real 1778626648] req@ffff954350196a00 x1865013828416384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626648.866205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626648.866216:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626648.866237:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626648.866247:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10983 00000100:00080000:1.0:1778626648.866251:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626648.866255:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626648.866261:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626648.866267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828416384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626648.866274:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828416896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626648.866304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626648.866317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626648.866325:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828416384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626648.866360:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626648.866398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626648.866431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626648/real 1778626648] req@ffff954350194e00 x1865013828416512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626648.866448:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828416512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626648.866458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626648.866463:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626648.866467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626648.866495:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626648.866538:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626648.866553:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626648/real 1778626648] req@ffff954350195880 x1865013828416640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626648.866569:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828416640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626648.866580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626648.866585:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626648.866589:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626648.866608:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626648.866639:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626648.866655:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626648/real 1778626648] req@ffff954350197b80 x1865013828416768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626648.866672:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828416768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626648.866684:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626648.866689:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626648.866694:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626648.866701:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626648.866728:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626648.866791:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626648/real 1778626648] req@ffff954350195180 x1865013828416896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626648.866809:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828416896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626648.866821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626648.866826:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626648.866831:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828416896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626649.057777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626649.057865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626649.825804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626650.081766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626650.081841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626650.849823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626651.105792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626651.105855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626651.873739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626652.129751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626652.129835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626652.897863:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626653.153767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626653.153827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626653.921805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626653.985779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626653.985791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626653.985811:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626653.985824:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10988 00000100:00080000:1.0:1778626653.985836:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626653.985844:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626653.985896:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828417024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626653.985921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626653.985924:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626653.985925:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626653.985944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10988 00000100:00080000:1.0:1778626653.985947:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626653.985949:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626653.985957:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626653.985968:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828417152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626653.985976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626653.985978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626653.985979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626653.985984:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10988 00000100:00080000:1.0:1778626653.985986:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626653.985989:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626653.986005:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626653.986013:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828417280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626653.986026:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828417024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626653.986026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626653.986039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626653.986042:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626653.986046:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10988 00000100:00080000:1.0:1778626653.986048:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626653.986051:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626653.986055:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626653.986065:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828417408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626653.986072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626653.986074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626653.986075:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626653.986078:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10988 00000100:00080000:1.0:1778626653.986080:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626653.986083:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626653.986086:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626653.986092:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828417536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626653.986118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626653.986174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626653/real 1778626653] req@ffff954350195f80 x1865013828417024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626653.986192:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828417024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626653.986203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626653.986206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626653.986210:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828417024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626653.986216:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626653.986234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626653.986242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626653/real 1778626653] req@ffff954350195180 x1865013828417152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626653.986261:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828417152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626653.986267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626653.986270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626653.986272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626653.986296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626653.986308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626653.986324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626653/real 1778626653] req@ffff954350197b80 x1865013828417280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626653.986346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828417280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626653.986352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626653.986355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626653.986357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626653.986362:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626653.986377:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626653.986384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626653/real 1778626653] req@ffff954350195880 x1865013828417408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626653.986393:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828417408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626653.986399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626653.986401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626653.986403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626653.986430:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626653.986446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626653.986453:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626653/real 1778626653] req@ffff954350194e00 x1865013828417536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626653.986462:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828417536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626653.986479:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626653.986482:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626653.986484:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626654.177759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626654.177875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626654.945768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626655.201773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626655.201834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626655.969789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626656.225732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626656.225843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626656.993733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626657.249764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626657.249859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626658.017769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626658.273770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626658.273860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626659.041747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626659.105801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626659.105809:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626659.105829:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626659.105837:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10993 00000100:00080000:1.0:1778626659.105847:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626659.105854:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626659.105890:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828417664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626659.105918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626659.105921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626659.105923:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626659.105926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10993 00000100:00080000:1.0:1778626659.105929:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626659.105932:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626659.105939:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626659.105950:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828417792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626659.105957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626659.105960:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626659.105961:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626659.105964:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10993 00000100:00080000:1.0:1778626659.105966:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626659.105968:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626659.105971:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626659.105979:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828417664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626659.105981:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828417920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626659.106004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626659.106007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626659.106008:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626659.106011:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10993 00000100:00080000:1.0:1778626659.106015:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626659.106018:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626659.106022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626659.106051:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828418048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626659.106056:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626659.106069:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626659.106072:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626659.106074:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626659.106080:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10993 00000100:00080000:1.0:1778626659.106084:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626659.106088:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626659.106094:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626659.106102:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626659/real 1778626659] req@ffff954350196a00 x1865013828417664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626659.106109:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828418176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626659.106148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828417664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626659.106158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626659.106161:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626659.106164:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828417664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626659.106179:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626659.106204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626659.106217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626659/real 1778626659] req@ffff954350194380 x1865013828417792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626659.106241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828417792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626659.106254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626659.106256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626659.106259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626659.106277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626659.106293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626659.106300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626659/real 1778626659] req@ffff954350197800 x1865013828417920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626659.106307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828417920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626659.106313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626659.106315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626659.106317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828417920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626659.106320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626659.106330:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626659.106336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626659/real 1778626659] req@ffff954350195c00 x1865013828418048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626659.106342:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828418048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626659.106347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626659.106349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626659.106356:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626659.106360:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626659.106372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626659.106378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626659/real 1778626659] req@ffff954350195500 x1865013828418176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626659.106385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828418176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626659.106391:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626659.106393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626659.106394:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626659.297756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626659.297832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626660.065778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626660.321757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626660.321843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626661.089802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626661.345847:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626661.345924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626662.113816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626662.369814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626662.369921:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626663.137764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626663.393772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626663.393859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626664.161795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626664.225759:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626664.225770:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626664.225775:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626664.225807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 10998 00000100:00080000:2.0:1778626664.225822:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626664.225831:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626664.225879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828418304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626664.225919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626664.225924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626664.225926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626664.225932:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10998 00000100:00080000:2.0:1778626664.225937:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626664.225942:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626664.225954:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626664.225969:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828418304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626664.225972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828418432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626664.225988:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626664.225992:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626664.225994:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626664.225998:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10998 00000100:00080000:2.0:1778626664.226002:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626664.226008:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626664.226014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626664.226028:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828418560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626664.226045:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626664.226058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626664.226062:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626664.226065:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626664.226071:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10998 00000100:00080000:2.0:1778626664.226077:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626664.226083:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626664.226089:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626664/real 1778626664] req@ffff954358789180 x1865013828418304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778626664.226090:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626664.226111:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828418688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626664.226113:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828418304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626664.226144:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626664.226145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626664.226153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778626664.226153:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626664.226156:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626664.226163:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 10998 00000100:00100000:0.0:1778626664.226167:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828418304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626664.226169:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626664.226176:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626664.226182:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626664.226197:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626664.226204:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828418816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626664.226221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626664.226233:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626664/real 1778626664] req@ffff954358789c00 x1865013828418432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626664.226242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828418432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626664.226248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626664.226251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626664.226253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626664.226257:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626664.226268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626664.226273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626664/real 1778626664] req@ffff95435878b800 x1865013828418560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626664.226280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828418560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626664.226286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626664.226288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626664.226290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626664.226293:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626664.226304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626664.226316:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626664/real 1778626664] req@ffff95435878a300 x1865013828418688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626664.226323:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828418688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626664.226329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626664.226330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626664.226332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626664.226343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626664.226355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626664.226360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626664/real 1778626664] req@ffff95435878ad80 x1865013828418816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626664.226368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828418816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626664.226373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626664.226375:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626664.226377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828418816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626664.417764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626664.417825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626665.185741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626665.441770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626665.441838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626666.209811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626666.465797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626666.465882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626667.233824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626667.489750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626667.489848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626668.257767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626668.513827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626668.513892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626669.281774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626669.345780:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626669.345790:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626669.345795:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626669.345807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11003 00000100:00080000:2.0:1778626669.345821:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626669.345828:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626669.345887:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828418944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626669.345917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626669.345920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626669.345922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626669.345925:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11003 00000100:00080000:2.0:1778626669.345928:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626669.345930:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626669.345940:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626669.345963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828419072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626669.345970:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626669.345973:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626669.345974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626669.345977:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11003 00000100:00080000:2.0:1778626669.345978:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626669.345981:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626669.345983:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626669.346012:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828419200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626669.346022:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626669.346025:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828418944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626669.346026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626669.346033:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626669.346047:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11003 00000100:00080000:2.0:1778626669.346051:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626669.346056:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626669.346061:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626669.346070:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828419328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626669.346077:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626669.346080:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626669.346081:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626669.346084:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11003 00000100:00080000:2.0:1778626669.346086:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626669.346089:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626669.346093:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626669.346103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626669.346110:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828419456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626669.346155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626669/real 1778626669] req@ffff954358789f80 x1865013828418944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626669.346170:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828418944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626669.346179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626669.346182:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626669.346185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828418944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626669.346191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626669.346207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626669.346214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626669/real 1778626669] req@ffff95435878a680 x1865013828419072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626669.346222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828419072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626669.346228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626669.346230:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626669.346232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626669.346272:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626669.346297:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626669.346312:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626669/real 1778626669] req@ffff95435878ad80 x1865013828419200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626669.346329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828419200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626669.346399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626669.346402:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626669.346405:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626669.346409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626669.346426:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626669.346434:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626669/real 1778626669] req@ffff95435878a300 x1865013828419328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626669.346442:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828419328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626669.346448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626669.346450:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626669.346452:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626669.346456:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626669.346465:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626669.346470:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626669/real 1778626669] req@ffff95435878b800 x1865013828419456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626669.346477:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828419456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626669.346482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626669.346484:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626669.346491:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626669.537786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626669.537878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626670.305782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778626670.561905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626670.562175:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626671.329742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626671.585852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626671.585954:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626672.353802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626672.609796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626672.609919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626673.377775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626673.633876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626673.633887:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626674.401767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626674.465746:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626674.465754:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626674.465758:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626674.465772:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11008 00000100:00080000:2.0:1778626674.465781:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626674.465787:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626674.465818:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828419584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626674.465840:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626674.465842:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626674.465844:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626674.465847:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11008 00000100:00080000:2.0:1778626674.465849:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626674.465851:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626674.465858:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626674.465867:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828419712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626674.465874:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626674.465876:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626674.465878:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626674.465880:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11008 00000100:00080000:2.0:1778626674.465883:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626674.465885:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626674.465892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828419584:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778626674.465902:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626674.465913:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828419840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626674.465920:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626674.465923:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626674.465925:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626674.465928:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11008 00000100:00080000:2.0:1778626674.465932:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626674.465934:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626674.465937:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626674.465944:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828419968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626674.465952:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626674.465953:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626674.465957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626674.465958:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626674.465961:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11008 00000100:00080000:2.0:1778626674.465964:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626674.465967:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626674.465971:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626674.465978:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828420096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626674.465991:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626674/real 1778626674] req@ffff954358789c00 x1865013828419584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626674.466007:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828419584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626674.466016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626674.466020:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626674.466023:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828419584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626674.466028:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626674.466042:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626674.466049:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626674/real 1778626674] req@ffff954358789180 x1865013828419712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626674.466057:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828419712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626674.466063:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626674.466065:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626674.466074:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626674.466093:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626674.466106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626674.466113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626674/real 1778626674] req@ffff954358788e00 x1865013828419840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626674.466120:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828419840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626674.466125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626674.466127:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626674.466129:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626674.466133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626674.466142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626674.466147:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626674/real 1778626674] req@ffff954358789500 x1865013828419968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626674.466154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828419968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626674.466159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626674.466161:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626674.466162:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828419968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626674.466165:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626674.466174:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626674.466185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626674/real 1778626674] req@ffff954358788a80 x1865013828420096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626674.466202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828420096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626674.466208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626674.466210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626674.466212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626674.657982:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626674.658083:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626675.425748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626675.681951:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626675.682117:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626676.449767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626676.705724:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626676.705812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626677.473839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626677.729784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626677.729848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626678.497800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626678.753804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626678.753877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626679.521822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626679.585848:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626679.585856:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626679.585860:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626679.585868:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11014 00000100:00080000:2.0:1778626679.585895:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626679.585902:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626679.585941:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828420224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626679.585973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626679.585976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626679.585978:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626679.585982:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11014 00000100:00080000:2.0:1778626679.585985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626679.585988:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626679.585998:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626679.586011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828420352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626679.586038:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626679.586041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626679.586042:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626679.586045:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11014 00000100:00080000:2.0:1778626679.586048:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626679.586057:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828420224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626679.586070:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626679.586074:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626679.586082:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828420480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626679.586088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626679.586090:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626679.586092:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626679.586095:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11014 00000100:00080000:2.0:1778626679.586097:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626679.586099:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626679.586102:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626679.586112:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828420608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626679.586119:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626679.586122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626679.586123:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626679.586125:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11014 00000100:00080000:2.0:1778626679.586127:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626679.586140:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626679.586144:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626679.586152:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828420736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626679.586165:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626679.586208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626679/real 1778626679] req@ffff954358789880 x1865013828420224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626679.586226:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828420224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626679.586235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626679.586239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626679.586242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828420224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626679.586270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626679.586295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626679.586325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626679/real 1778626679] req@ffff954358788700 x1865013828420352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626679.586339:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828420352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626679.586349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626679.586353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626679.586357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626679.586365:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626679.586387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626679.586396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626679/real 1778626679] req@ffff95435878aa00 x1865013828420480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626679.586404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828420480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626679.586410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626679.586413:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626679.586415:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626679.586420:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626679.586433:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626679.586438:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626679/real 1778626679] req@ffff95435878b480 x1865013828420608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626679.586454:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828420608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626679.586466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626679.586469:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626679.586471:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626679.586476:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626679.586496:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626679.586507:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626679/real 1778626679] req@ffff954358788380 x1865013828420736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626679.586520:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828420736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626679.586530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626679.586534:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626679.586537:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626679.777812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626679.777984:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626680.545801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626680.801822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626680.801922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626681.569776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626681.825767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626681.825931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626682.593812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626682.849792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626682.849894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626683.617836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626683.873765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626683.873844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626684.641782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626684.705798:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626684.705817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626684.705822:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626684.705832:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11019 00000100:00080000:1.0:1778626684.705846:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626684.705852:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626684.705894:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828420864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626684.705922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626684.705925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626684.705926:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626684.705929:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11019 00000100:00080000:1.0:1778626684.705932:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626684.705935:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626684.705944:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626684.705954:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828420992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626684.705962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626684.705979:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828420864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626684.705979:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626684.705982:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626684.705988:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11019 00000100:00080000:1.0:1778626684.705998:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626684.706003:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626684.706007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626684.706023:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828421120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626684.706043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626684.706047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626684.706049:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626684.706052:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11019 00000100:00080000:1.0:1778626684.706062:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626684.706066:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626684.706073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626684.706075:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626684.706089:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828421248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626684.706098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626684.706100:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626684.706102:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626684.706108:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11019 00000100:00080000:1.0:1778626684.706113:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626684.706115:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626684/real 1778626684] req@ffff954350194a80 x1865013828420864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626684.706124:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626684.706137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828420864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778626684.706142:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778626684.706158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626684.706164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778626684.706164:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828421376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626684.706175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828420864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626684.706214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626684.706244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626684.706253:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626684/real 1778626684] req@ffff954350194700 x1865013828420992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626684.706261:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828420992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626684.706267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626684.706270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626684.706272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828420992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626684.706276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626684.706287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626684.706292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626684/real 1778626684] req@ffff954350197480 x1865013828421120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626684.706300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828421120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626684.706304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626684.706306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626684.706308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626684.706317:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626684.706331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626684.706345:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626684/real 1778626684] req@ffff954350196d80 x1865013828421248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626684.706353:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828421248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626684.706358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626684.706360:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626684.706362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626684.706366:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626684.706376:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626684.706381:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626684/real 1778626684] req@ffff954350194000 x1865013828421376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626684.706388:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828421376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626684.706393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626684.706395:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626684.706397:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626684.897742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626684.897938:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626685.665764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626685.921820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626685.921892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626686.689817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626686.945728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626686.945808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626687.713736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626687.969769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626687.969825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626688.737758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626688.993718:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626688.993811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626689.761749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626689.825766:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626689.825791:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626689.825794:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626689.825802:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11024 00000100:00080000:3.0:1778626689.825812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626689.825817:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626689.825845:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828421504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626689.825872:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626689.825875:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626689.825877:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626689.825882:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11024 00000100:00080000:3.0:1778626689.825886:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626689.825890:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626689.825899:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626689.825912:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828421632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626689.825920:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626689.825922:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626689.825924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626689.825942:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11024 00000100:00080000:3.0:1778626689.825946:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626689.825949:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626689.825954:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626689.825958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828421504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626689.825966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828421760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626689.825982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626689.825984:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626689.825986:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626689.825989:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11024 00000100:00080000:3.0:1778626689.825993:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626689.825995:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626689.825999:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626689.826008:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828421888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626689.826014:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626689.826017:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626689.826018:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626689.826022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626689.826025:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11024 00000100:00080000:3.0:1778626689.826028:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626689.826042:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626689.826045:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626689.826053:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828422016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626689.826060:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626689/real 1778626689] req@ffff95434701f800 x1865013828421504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626689.826081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828421504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626689.826091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626689.826096:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626689.826100:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828421504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626689.826108:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626689.826141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626689.826154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626689/real 1778626689] req@ffff95434701d880 x1865013828421632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626689.826166:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828421632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626689.826176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626689.826179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626689.826182:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626689.826205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626689.826225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626689.826236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626689/real 1778626689] req@ffff95434701ea00 x1865013828421760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626689.826270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828421760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626689.826278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626689.826281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626689.826285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626689.826292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626689.826315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626689.826326:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626689/real 1778626689] req@ffff95434701c000 x1865013828421888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626689.826338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828421888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626689.826346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626689.826349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626689.826352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828421888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626689.826358:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626689.826378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626689.826388:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626689/real 1778626689] req@ffff95434701d500 x1865013828422016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626689.826400:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828422016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626689.826422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626689.826425:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626689.826429:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626690.017810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626690.017875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626690.785758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626691.041810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626691.041848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626691.809802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626692.065788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626692.065839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626692.833811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626693.089816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626693.089909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626693.857789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626694.113921:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626694.113941:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626694.881796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626694.945769:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626694.945778:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626694.945782:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626694.945792:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11029 00000100:00080000:2.0:1778626694.945810:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626694.945818:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626694.945871:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828422144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626694.945894:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626694.945896:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626694.945898:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626694.945901:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11029 00000100:00080000:2.0:1778626694.945904:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626694.945908:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626694.945917:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626694.945931:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828422272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626694.945964:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626694.945967:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626694.945969:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626694.945972:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11029 00000100:00080000:2.0:1778626694.945976:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626694.945981:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626694.945988:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828422144:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778626694.945991:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626694.946007:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828422400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626694.946017:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626694.946020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626694.946021:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626694.946024:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11029 00000100:00080000:2.0:1778626694.946027:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626694.946029:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626694.946035:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626694.946048:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828422528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626694.946058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626694.946062:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626694.946064:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626694.946068:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11029 00000100:00080000:2.0:1778626694.946081:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626694.946089:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626694.946091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778626694.946099:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626694.946114:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828422656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626694.946149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626694/real 1778626694] req@ffff95435878b100 x1865013828422144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626694.946167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828422144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626694.946176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626694.946180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626694.946184:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828422144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626694.946190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626694.946207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626694.946215:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626694/real 1778626694] req@ffff95435878bb80 x1865013828422272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626694.946223:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828422272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626694.946242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626694.946245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626694.946247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626694.946273:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626694.946286:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626694.946293:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626694/real 1778626694] req@ffff954358788380 x1865013828422400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626694.946300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828422400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626694.946306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626694.946308:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626694.946310:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626694.946314:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626694.946324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626694.946329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626694/real 1778626694] req@ffff95435878b480 x1865013828422528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626694.946336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828422528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626694.946341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626694.946344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626694.946350:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626694.946355:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626694.946368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626694.946374:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626694/real 1778626694] req@ffff95435878aa00 x1865013828422656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626694.946382:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828422656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626694.946388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626694.946390:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626694.946392:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626695.137742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626695.137949:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626695.905778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626696.161779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626696.161865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626696.929783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626697.185730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626697.185824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626697.953745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626698.209755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626698.209824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626698.977826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626699.233837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626699.233916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626700.001745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626700.065802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626700.065810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626700.065814:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626700.065823:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11034 00000100:00080000:1.0:1778626700.065834:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626700.065839:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626700.065882:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828422784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626700.065904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626700.065907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626700.065921:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626700.065925:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11034 00000100:00080000:1.0:1778626700.065929:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626700.065932:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626700.065941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626700.065948:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828422784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626700.065954:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828422912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626700.065963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626700.065965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626700.065966:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626700.065969:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11034 00000100:00080000:1.0:1778626700.065971:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626700.065973:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626700.065977:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626700.065984:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828423040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626700.065991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626700.065993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626700.065995:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626700.065997:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11034 00000100:00080000:1.0:1778626700.065999:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626700.066001:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626700.066006:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778626700.066007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626700.066022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828423168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626700.066049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626700.066053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778626700.066056:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626700/real 1778626700] req@ffff954350197100 x1865013828422784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626700.066057:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626700.066067:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11034 00000100:00080000:1.0:1778626700.066074:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626700.066079:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828422784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626700.066083:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626700.066094:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778626700.066094:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778626700.066099:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778626700.066107:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828423296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626700.066124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828422784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626700.066150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626700.066169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626700.066177:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626700/real 1778626700] req@ffff954350194000 x1865013828422912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626700.066186:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828422912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626700.066191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626700.066194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626700.066196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828422912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626700.066199:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626700.066209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626700.066215:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626700/real 1778626700] req@ffff954350196d80 x1865013828423040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626700.066222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828423040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626700.066226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626700.066228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626700.066230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626700.066233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626700.066243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626700.066255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626700/real 1778626700] req@ffff954350197480 x1865013828423168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626700.066270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828423168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626700.066275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626700.066277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626700.066279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626700.066283:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626700.066294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626700.066300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626700/real 1778626700] req@ffff954350194700 x1865013828423296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626700.066307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828423296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626700.066312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626700.066314:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626700.066316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626700.257771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626700.257865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626701.025793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626701.281783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626701.281845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626702.049762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626702.305778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626702.305835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626703.073782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626703.329796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626703.329893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626704.097762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626704.353738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626704.353806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626705.121799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626705.185813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626705.185826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626705.185830:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626705.185838:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11039 00000100:00080000:1.0:1778626705.185849:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626705.185857:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626705.185895:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828423424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626705.185919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626705.185937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626705.185940:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626705.185944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11039 00000100:00080000:1.0:1778626705.185949:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626705.185954:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626705.185965:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626705.185969:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828423424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626705.185982:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828423552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626705.185995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626705.185999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626705.186001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626705.186004:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11039 00000100:00080000:1.0:1778626705.186016:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626705.186018:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626705.186022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626705.186032:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828423680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626705.186039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626705.186041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626705.186043:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626705.186046:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11039 00000100:00080000:1.0:1778626705.186049:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626705.186052:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626705.186060:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423552:192.168.204.102@tcp:38:kworker.0 00000008:00100000:1.0:1778626705.186067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626705.186083:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828423808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626705.186093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626705.186095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626705.186097:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626705.186100:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11039 00000100:00080000:1.0:1778626705.186103:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626705.186106:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626705.186110:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626705.186112:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778626705.186121:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828423936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626705.186147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626705.186165:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626705.377750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626705.377804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626706.145770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626706.401741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626706.401799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626707.169775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626707.425769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626707.425820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626708.193739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778626708.257902:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778626708.257938:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778626708.257953:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626708.257992:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626708.258002:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626708.258011:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626708.258021:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626708.258054:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626705/real 1778626708] req@ffff954350197800 x1865013828423808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626708.258081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828423808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626708.258093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626708.258098:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626708.258103:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626708.258111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626705/real 1778626708] req@ffff954350194380 x1865013828423936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626708.258123:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828423936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626708.258131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626708.258134:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626708.258137:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626708.258143:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626705/real 1778626708] req@ffff954350195500 x1865013828423552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626708.258170:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828423552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626708.258178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626708.258182:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626708.258186:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626708.258197:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626705/real 1778626708] req@ffff954350195c00 x1865013828423680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626708.258210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828423680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626708.258217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626708.258220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626708.258223:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828423680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626708.258230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626705/real 1778626708] req@ffff954350194a80 x1865013828423424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626708.258242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828423424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626708.258250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626708.258253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626708.258256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828423424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626708.449742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626708.449796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626709.217770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626709.473760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626709.473853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626710.241758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626710.305828:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626710.305839:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626710.305844:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626710.305855:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11044 00000100:00080000:2.0:1778626710.305866:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626710.305873:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626710.305908:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828424064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626710.305937:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626710.305941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626710.305943:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626710.305947:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11044 00000100:00080000:2.0:1778626710.305967:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626710.305973:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626710.305983:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626710.305997:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828424064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626710.306001:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828424192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626710.306025:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626710.306029:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626710.306031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626710.306036:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11044 00000100:00080000:2.0:1778626710.306040:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626710.306045:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626710.306050:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626710.306063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828424320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626710.306074:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626710.306076:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626710.306081:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626710.306083:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626710.306089:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11044 00000100:00080000:2.0:1778626710.306094:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626710.306098:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626710.306104:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626710.306119:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828424448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626710.306121:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626710/real 1778626710] req@ffff954358788700 x1865013828424064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626710.306158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828424064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626710.306169:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626710.306174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626710.306174:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626710.306178:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626710.306183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778626710.306187:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11044 00000100:00100000:0.0:1778626710.306194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828424064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626710.306194:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626710.306204:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626710.306214:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626710.306232:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626710.306233:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828424576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626710.306271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626710.306287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626710/real 1778626710] req@ffff954358789880 x1865013828424192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626710.306302:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828424192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626710.306312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626710.306316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626710.306319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626710.306326:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626710.306350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626710.306362:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626710/real 1778626710] req@ffff954358788a80 x1865013828424320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626710.306395:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828424320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626710.306404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626710.306407:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626710.306412:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626710.306419:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626710.306444:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626710.306456:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626710/real 1778626710] req@ffff954358789500 x1865013828424448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626710.306468:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828424448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626710.306476:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626710.306480:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626710.306483:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626710.306502:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626710.306540:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626710.306554:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626710/real 1778626710] req@ffff954358788e00 x1865013828424576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626710.306566:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828424576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626710.306589:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626710.306593:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626710.306597:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626710.497742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626710.497816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626711.265768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626711.521840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626711.521895:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626712.289783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626712.545746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626712.545837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626713.313785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626713.569763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626713.569891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626714.337774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626714.593735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626714.593815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626715.361758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626715.425796:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626715.425804:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626715.425808:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626715.425816:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11049 00000100:00080000:3.0:1778626715.425828:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626715.425835:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626715.425873:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828424704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626715.425896:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626715.425899:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626715.425920:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626715.425924:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11049 00000100:00080000:3.0:1778626715.425928:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626715.425931:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626715.425949:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626715.425958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828424704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626715.425963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828424832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626715.425972:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626715.425974:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626715.425976:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626715.425978:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11049 00000100:00080000:3.0:1778626715.425981:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626715.425985:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626715.425991:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626715.426003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828424960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626715.426013:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626715.426014:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626715.426020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626715.426022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626715.426029:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11049 00000100:00080000:3.0:1778626715.426034:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626715.426039:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626715.426045:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626715.426051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626715/real 1778626715] req@ffff95434701c700 x1865013828424704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626715.426067:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828425088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626715.426070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828424704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626715.426095:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626715.426096:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778626715.426100:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626715.426102:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626715.426103:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778626715.426111:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11049 00000100:00100000:0.0:1778626715.426112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828424704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626715.426115:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626715.426122:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626715.426125:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626715.426132:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828425216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626715.426144:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626715.426171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626715.426190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626715/real 1778626715] req@ffff95434701f100 x1865013828424832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626715.426204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828424832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626715.426231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626715.426235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626715.426239:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626715.426246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626715.426269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626715.426280:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626715/real 1778626715] req@ffff95434701c380 x1865013828424960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626715.426293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828424960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626715.426312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626715.426315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626715.426319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828424960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626715.426325:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626715.426348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626715.426359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626715/real 1778626715] req@ffff95434701d500 x1865013828425088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626715.426371:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828425088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626715.426379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626715.426383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626715.426393:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626715.426409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626715.426449:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626715.426460:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626715/real 1778626715] req@ffff95434701c000 x1865013828425216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626715.426470:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828425216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626715.426485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626715.426488:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626715.426490:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626715.617732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626715.617807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626716.385751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626716.641758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626716.641874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626717.409769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626717.665769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626717.665888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626718.433774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626718.689754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626718.689834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626719.457759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626719.713756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626719.713826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626720.481785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626720.545789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626720.545800:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626720.545805:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626720.545815:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11055 00000100:00080000:1.0:1778626720.545825:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626720.545833:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626720.545869:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828425344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626720.545901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626720.545904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626720.545905:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626720.545909:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11055 00000100:00080000:1.0:1778626720.545911:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626720.545915:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626720.545922:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626720.545934:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828425472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626720.545942:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626720.545945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626720.545946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626720.545950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828425344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626720.545952:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11055 00000100:00080000:1.0:1778626720.545957:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626720.545960:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626720.545964:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626720.545988:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828425600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626720.545995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626720.545998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626720.545999:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626720.546002:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11055 00000100:00080000:1.0:1778626720.546005:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778626720.546009:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626720.546011:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626720.546016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626720.546025:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828425728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626720.546031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626720.546034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626720.546036:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626720.546039:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11055 00000100:00080000:1.0:1778626720.546042:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626720.546045:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626720/real 1778626720] req@ffff954350196a00 x1865013828425344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626720.546047:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626720.546052:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626720.546062:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828425856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626720.546070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828425344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626720.546082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626720.546087:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626720.546092:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828425344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626720.546123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626720.546156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626720.546167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626720/real 1778626720] req@ffff954350194e00 x1865013828425472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626720.546185:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828425472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626720.546195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626720.546199:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626720.546203:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626720.546210:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626720.546243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626720.546255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626720/real 1778626720] req@ffff954350195880 x1865013828425600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626720.546269:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828425600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626720.546277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626720.546280:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626720.546283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626720.546289:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626720.546336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626720.546348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626720/real 1778626720] req@ffff954350197b80 x1865013828425728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626720.546361:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828425728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626720.546383:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626720.546387:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626720.546390:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626720.546397:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626720.546420:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626720.546432:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626720/real 1778626720] req@ffff954350195180 x1865013828425856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626720.546442:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828425856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626720.546451:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626720.546454:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626720.546458:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828425856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626720.737779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626720.737839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626721.505765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626721.761755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626721.761802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626722.529763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626722.785768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626722.785887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626723.553754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626723.809807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626723.809902:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626724.577766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626724.833785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626724.833863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626725.601754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626725.665787:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626725.665800:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626725.665823:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626725.665836:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11060 00000100:00080000:3.0:1778626725.665850:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626725.665882:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626725.665941:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828425984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626725.665969:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626725.665972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626725.665974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626725.665978:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11060 00000100:00080000:3.0:1778626725.665982:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626725.665984:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626725.666000:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626725.666017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828426112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626725.666018:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828425984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626725.666035:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626725.666037:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626725.666039:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626725.666042:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11060 00000100:00080000:3.0:1778626725.666045:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626725.666047:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626725.666051:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626725.666060:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828426240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626725.666067:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626725.666071:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626725.666078:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626725.666082:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11060 00000100:00080000:3.0:1778626725.666086:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626725.666090:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626725.666094:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626725.666118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626725.666122:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828426368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626725.666130:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626725.666133:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626725.666134:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626725.666146:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11060 00000100:00080000:3.0:1778626725.666149:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626725.666152:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626725.666155:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626725.666162:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828426496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626725.666175:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626725/real 1778626725] req@ffff95434701ea00 x1865013828425984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626725.666195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828425984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626725.666205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626725.666208:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626725.666212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828425984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626725.666267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626725.666287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626725.666297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626725/real 1778626725] req@ffff95434701d880 x1865013828426112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626725.666307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828426112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626725.666313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626725.666317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626725.666320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626725.666326:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626725.666345:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626725.666356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626725/real 1778626725] req@ffff95434701f800 x1865013828426240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626725.666370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828426240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626725.666379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626725.666382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626725.666385:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626725.666390:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626725.666407:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626725.666415:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626725/real 1778626725] req@ffff95434701df80 x1865013828426368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626725.666430:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828426368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626725.666436:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626725.666438:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626725.666440:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626725.666444:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626725.666459:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626725.666465:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626725/real 1778626725] req@ffff95434701ca80 x1865013828426496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626725.666484:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828426496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626725.666489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626725.666491:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626725.666493:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626725.857795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626725.857943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626726.625784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626726.881740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626726.881834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626727.649776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626727.905747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626727.905861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626728.673768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626728.929760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626728.929831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626729.697759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626729.953747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626729.953833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626730.721745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626730.785835:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626730.785848:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626730.785853:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626730.785866:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11065 00000100:00080000:3.0:1778626730.785923:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626730.785935:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626730.785999:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828426624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626730.786058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626730.786074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626730.786077:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626730.786083:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11065 00000100:00080000:3.0:1778626730.786105:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626730.786109:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626730.786142:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626730.786183:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828426752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626730.786193:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626730.786197:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626730.786200:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626730.786214:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11065 00000100:00080000:3.0:1778626730.786218:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626730.786222:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626730.786228:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626730.786237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828426624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626730.786239:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828426880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626730.786256:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626730.786267:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626730.786269:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626730.786273:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11065 00000100:00080000:3.0:1778626730.786278:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626730.786283:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626730.786289:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626730.786302:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828427008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626730.786311:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626730.786316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626730.786318:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626730.786321:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626730.786325:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11065 00000100:00080000:3.0:1778626730.786330:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626730.786335:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626730.786350:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626730.786360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626730/real 1778626730] req@ffff95434701f480 x1865013828426624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626730.786366:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828427136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626730.786404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828426624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626730.786432:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626730.786439:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626730.786445:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828426624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626730.786456:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626730.786491:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626730.786509:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626730/real 1778626730] req@ffff95434701e680 x1865013828426752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626730.786528:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828426752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626730.786540:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626730.786543:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626730.786547:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626730.786571:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626730.786592:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626730.786601:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626730/real 1778626730] req@ffff95434701d180 x1865013828426880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626730.786632:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828426880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626730.786641:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626730.786645:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626730.786648:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828426880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626730.786665:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626730.786688:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626730.786743:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626730/real 1778626730] req@ffff95434701ca80 x1865013828427008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626730.786758:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828427008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626730.786784:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626730.786787:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626730.786791:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626730.786797:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626730.786818:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626730.786844:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626730/real 1778626730] req@ffff95434701df80 x1865013828427136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626730.786854:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828427136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626730.786861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626730.786863:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626730.786866:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626730.977770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626730.977836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626731.745750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626732.001802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626732.001905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626732.769779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626733.025858:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626733.025979:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626733.793800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626734.049749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626734.049813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626734.817790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626735.073780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626735.073844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626735.841818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626735.905727:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626735.905735:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626735.905750:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626735.905757:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11070 00000100:00080000:3.0:1778626735.905767:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626735.905772:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626735.905800:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828427264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626735.905822:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626735.905825:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626735.905826:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626735.905829:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11070 00000100:00080000:3.0:1778626735.905832:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626735.905834:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626735.905841:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626735.905849:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828427392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626735.905855:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626735.905858:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626735.905859:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626735.905861:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11070 00000100:00080000:3.0:1778626735.905862:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626735.905864:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626735.905868:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626735.905876:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828427520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626735.905898:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626735.905903:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1778626735.905908:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828427264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626735.905909:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626735.905924:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11070 00000100:00080000:3.0:1778626735.905929:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626735.905936:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626735.905943:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626735.905963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828427648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626735.905969:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626735.905972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626735.905973:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626735.905975:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11070 00000100:00080000:3.0:1778626735.905981:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626735.905985:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626735.905989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626735.905995:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828427776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626735.906000:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626735.906085:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626735/real 1778626735] req@ffff95434701f800 x1865013828427264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626735.906123:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828427264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626735.906135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626735.906139:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626735.906159:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828427264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626735.906166:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626735.906187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626735.906196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626735/real 1778626735] req@ffff95434701d880 x1865013828427392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626735.906205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828427392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626735.906211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626735.906213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626735.906215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626735.906240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626735.906250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626735.906256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626735/real 1778626735] req@ffff95434701ea00 x1865013828427520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626735.906264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828427520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626735.906278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626735.906280:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626735.906282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626735.906287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626735.906301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626735.906307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626735/real 1778626735] req@ffff95434701c000 x1865013828427648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626735.906314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828427648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626735.906320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626735.906321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626735.906323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626735.906327:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626735.906337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626735.906343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626735/real 1778626735] req@ffff95434701d500 x1865013828427776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626735.906350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828427776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626735.906355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626735.906357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626735.906363:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828427776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626736.097748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626736.097857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626736.865767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626737.122130:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626737.122272:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626737.889747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626738.145964:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626738.146029:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626738.913776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626739.169777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626739.170122:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626739.937748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626740.193868:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626740.194062:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626740.961727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626741.025760:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626741.025767:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626741.025787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626741.025795:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11075 00000100:00080000:3.0:1778626741.025805:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626741.025810:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626741.025839:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828427904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626741.025864:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626741.025867:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626741.025868:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626741.025872:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11075 00000100:00080000:3.0:1778626741.025874:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626741.025876:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626741.025884:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626741.025892:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828428032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626741.025899:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626741.025901:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626741.025903:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626741.025906:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11075 00000100:00080000:3.0:1778626741.025908:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626741.025911:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626741.025913:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626741.025935:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828428160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626741.025939:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828427904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626741.025958:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626741.025960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626741.025962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626741.025965:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11075 00000100:00080000:3.0:1778626741.025970:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626741.025973:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626741.025978:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626741.025989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828428288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626741.025996:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626741.025998:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626741.026000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626741.026003:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626741.026006:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11075 00000100:00080000:3.0:1778626741.026009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626741.026011:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626741.026015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626741.026023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828428416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626741.026037:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626741/real 1778626741] req@ffff95434701c380 x1865013828427904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626741.026052:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828427904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626741.026061:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626741.026065:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626741.026068:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828427904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626741.026073:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626741.026086:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626741.026094:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626741/real 1778626741] req@ffff95434701f100 x1865013828428032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626741.026101:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828428032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626741.026107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626741.026109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626741.026111:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626741.026138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626741.026152:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626741.026158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626741/real 1778626741] req@ffff95434701c700 x1865013828428160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626741.026166:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828428160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626741.026171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626741.026173:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626741.026175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626741.026178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626741.026189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626741.026193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626741/real 1778626741] req@ffff95434701d500 x1865013828428288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626741.026200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828428288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626741.026205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626741.026207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626741.026209:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626741.026212:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626741.026221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626741.026226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626741/real 1778626741] req@ffff95434701c000 x1865013828428416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626741.026239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828428416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626741.026245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626741.026247:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626741.026249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428416:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626741.217890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626741.217964:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778626741.985776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626742.241757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626742.241946:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626743.009828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626743.265778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626743.265836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626744.033776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626744.289772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626744.289840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626745.057762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626745.313737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626745.313887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626746.081785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626746.145872:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626746.145881:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626746.145885:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626746.145894:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11080 00000100:00080000:0.0:1778626746.145904:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626746.145911:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626746.145945:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828428544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626746.145991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626746.145995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626746.145998:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626746.146003:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11080 00000100:00080000:0.0:1778626746.146008:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626746.146013:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626746.146039:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626746.146042:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828428544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626746.146053:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828428672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626746.146064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626746.146067:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626746.146069:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626746.146072:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11080 00000100:00080000:0.0:1778626746.146076:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626746.146080:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626746.146083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626746.146092:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828428800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626746.146099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626746.146105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626746.146106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626746.146109:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626746.146122:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11080 00000100:00080000:0.0:1778626746.146125:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626746.146128:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626746.146131:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626746.146139:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828428928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626746.146146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626746.146149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626746.146155:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626746.146159:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11080 00000100:00080000:0.0:1778626746.146163:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626746.146166:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626746/real 1778626746] req@ffff954342f4bb80 x1865013828428544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626746.146169:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626746.146186:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626746.146200:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828429056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626746.146206:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828428544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626746.146223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626746.146228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626746.146232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828428544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626746.146265:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626746.146293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626746.146303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626746/real 1778626746] req@ffff954342f49880 x1865013828428672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626746.146312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828428672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626746.146319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626746.146321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626746.146323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626746.146327:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626746.146338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626746.146343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626746/real 1778626746] req@ffff954342f4a300 x1865013828428800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626746.146351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828428800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626746.146368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626746.146370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626746.146372:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626746.146376:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626746.146386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626746.146391:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626746/real 1778626746] req@ffff954342f4b480 x1865013828428928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626746.146410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828428928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626746.146416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626746.146418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626746.146420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828428928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626746.146434:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626746.146446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626746.146452:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626746/real 1778626746] req@ffff954342f49c00 x1865013828429056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626746.146460:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828429056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626746.146466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626746.146468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626746.146470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626746.337833:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626746.337907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626747.105781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626747.361759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626747.361879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626748.129758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626748.385806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626748.385896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626749.153825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626749.409807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626749.409922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626750.177889:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626750.433850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626750.433900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626751.201863:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626751.265878:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626751.265893:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626751.265900:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626751.265915:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11085 00000100:00080000:0.0:1778626751.265932:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626751.265960:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626751.266029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828429184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626751.266084:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626751.266092:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626751.266095:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626751.266104:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11085 00000100:00080000:0.0:1778626751.266112:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626751.266158:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626751.266175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626751.266183:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828429184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626751.266211:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828429312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626751.266237:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626751.266245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626751.266249:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626751.266258:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11085 00000100:00080000:0.0:1778626751.266269:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778626751.266278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626751.266286:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626751.266298:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626751.266342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828429440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626751.266363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626751/real 1778626751] req@ffff954342f49500 x1865013828429184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626751.266364:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626751.266370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626751.266374:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626751.266384:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11085 00000100:00080000:0.0:1778626751.266396:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626751.266410:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626751.266421:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626751.266430:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828429184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626751.266463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778626751.266465:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828429568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626751.266475:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626751.266487:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828429184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626751.266502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626751.266521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626751.266527:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626751.266527:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429312:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778626751.266544:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11085 00000100:00080000:0.0:1778626751.266554:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626751.266562:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626751.266589:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778626751.266589:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626751.266611:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626751/real 1778626751] req@ffff954342f4ad80 x1865013828429312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626751.266622:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828429696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626751.266638:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828429312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626751.266663:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626751.266669:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626751.266675:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626751.266683:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626751.266790:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626751.266805:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626751/real 1778626751] req@ffff954342f48a80 x1865013828429440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626751.266819:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828429440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626751.266844:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626751.266848:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626751.266851:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626751.266858:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626751.266883:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626751.266893:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626751/real 1778626751] req@ffff954342f4b100 x1865013828429568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626751.266905:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828429568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626751.266916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626751.266921:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626751.266927:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626751.266956:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626751.266989:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626751.267008:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626751/real 1778626751] req@ffff954342f49f80 x1865013828429696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626751.267047:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828429696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626751.267058:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626751.267062:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626751.267076:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626751.457838:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626751.457952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626752.225765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626752.481800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626752.481864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626753.249836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626753.505775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626753.505887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626754.273792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626754.529799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626754.529918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626755.297835:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626755.553799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626755.553907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626756.321730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626756.385761:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626756.385769:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626756.385773:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626756.385781:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11090 00000100:00080000:3.0:1778626756.385801:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626756.385806:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626756.385833:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828429824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626756.385860:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626756.385864:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626756.385866:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626756.385871:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11090 00000100:00080000:3.0:1778626756.385875:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626756.385880:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626756.385891:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626756.385900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828429824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626756.385905:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828429952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626756.385914:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626756.385916:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626756.385918:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626756.385920:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11090 00000100:00080000:3.0:1778626756.385923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626756.385926:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626756.385929:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626756.385936:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828430080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626756.385943:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626756.385950:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626756.385951:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626756.385954:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11090 00000100:00080000:3.0:1778626756.385958:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626756.385960:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626756.385965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626756.385968:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626756.385976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828430208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626756.385984:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626756.385987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626756.385988:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626756.385991:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11090 00000100:00080000:3.0:1778626756.385994:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626756.385996:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626756.385999:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626756.386001:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626756/real 1778626756] req@ffff95434701ea00 x1865013828429824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626756.386008:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828430336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626756.386024:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828429824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626756.386033:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626756.386036:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626756.386039:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828429824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626756.386056:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626756.386070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626756.386077:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626756/real 1778626756] req@ffff95434701d880 x1865013828429952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626756.386085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828429952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626756.386091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626756.386093:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626756.386102:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828429952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626756.386106:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626756.386119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626756.386125:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626756/real 1778626756] req@ffff95434701f800 x1865013828430080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626756.386132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828430080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626756.386137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626756.386139:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626756.386141:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626756.386144:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626756.386153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626756.386158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626756/real 1778626756] req@ffff95434701df80 x1865013828430208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626756.386165:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828430208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626756.386170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626756.386172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626756.386174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626756.386177:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626756.386185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626756.386196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626756/real 1778626756] req@ffff95434701ca80 x1865013828430336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626756.386203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828430336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626756.386209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626756.386211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626756.386213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626756.577770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626756.577888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626757.345769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626757.601739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626757.601821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626758.369807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626758.625765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626758.625879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626759.393792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626759.649769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626759.649824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626760.417738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626760.673760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626760.673832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626761.441814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626761.505905:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626761.505941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626761.505946:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626761.505957:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11096 00000100:00080000:2.0:1778626761.505970:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626761.505977:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626761.506031:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828430464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626761.506069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626761.506074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626761.506076:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626761.506081:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11096 00000100:00080000:2.0:1778626761.506085:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626761.506111:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626761.506126:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626761.506139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828430464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626761.506151:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828430592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626761.506173:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626761.506179:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626761.506183:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626761.506190:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11096 00000100:00080000:2.0:1778626761.506201:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626761.506207:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778626761.506216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778626761.506220:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626761.506242:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828430720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626761.506256:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626761.506261:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626761/real 1778626761] req@ffff954358789180 x1865013828430464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626761.506262:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626761.506268:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626761.506283:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11096 00000100:00080000:2.0:1778626761.506292:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626761.506315:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626761.506324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828430464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778626761.506327:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626761.506350:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828430848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626761.506355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626761.506372:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626761.506376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778626761.506387:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626761.506390:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626761.506405:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11096 00000100:00100000:0.0:1778626761.506413:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828430464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626761.506414:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626761.506428:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626761.506436:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626761.506464:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828430976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626761.506483:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626761.506568:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626761.506598:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626761/real 1778626761] req@ffff954358789c00 x1865013828430592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626761.506615:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828430592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626761.506627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626761.506631:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626761.506636:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626761.506644:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626761.506666:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626761.506712:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626761/real 1778626761] req@ffff95435878b800 x1865013828430720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626761.506726:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828430720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626761.506736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626761.506740:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626761.506763:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626761.506771:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626761.506794:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626761.506816:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626761/real 1778626761] req@ffff95435878a300 x1865013828430848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626761.506829:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828430848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626761.506855:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626761.506859:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626761.506863:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626761.506886:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626761.506909:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626761.506920:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626761/real 1778626761] req@ffff95435878ad80 x1865013828430976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626761.506934:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828430976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626761.506945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626761.506948:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626761.506952:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828430976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626761.697849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626761.697954:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626762.465798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626762.721733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626762.721853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626763.489821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626763.745808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626763.745906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626764.513804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626764.769793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626764.770001:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626765.537759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626765.793772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626765.793828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626766.561732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626766.625741:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626766.625749:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626766.625753:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626766.625761:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11101 00000100:00080000:2.0:1778626766.625771:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626766.625807:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626766.625839:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828431104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626766.625863:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626766.625866:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626766.625867:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626766.625871:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11101 00000100:00080000:2.0:1778626766.625888:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626766.625891:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626766.625899:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626766.625911:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828431232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626766.625913:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828431104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626766.625927:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626766.625930:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626766.625931:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626766.625934:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11101 00000100:00080000:2.0:1778626766.625936:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626766.625939:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626766.625942:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626766.625951:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828431360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626766.625957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626766.625959:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626766.625960:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626766.625963:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11101 00000100:00080000:2.0:1778626766.625966:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626766.625968:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626766.625971:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626766.625975:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626766.625981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828431488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626766.626001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626766.626004:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626766.626005:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626766.626007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11101 00000100:00080000:2.0:1778626766.626011:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626766.626014:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626766.626018:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626766.626022:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626766/real 1778626766] req@ffff95435878a680 x1865013828431104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626766.626029:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828431616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626766.626045:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828431104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626766.626053:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626766.626056:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626766.626059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828431104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626766.626079:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626766.626094:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626766.626102:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626766/real 1778626766] req@ffff954358789f80 x1865013828431232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626766.626110:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828431232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626766.626116:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626766.626118:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626766.626120:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626766.626123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626766.626132:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626766.626137:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626766/real 1778626766] req@ffff95435878ad80 x1865013828431360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626766.626144:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828431360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626766.626158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626766.626161:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626766.626163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626766.626166:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626766.626180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626766.626186:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626766/real 1778626766] req@ffff95435878a300 x1865013828431488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626766.626193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828431488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626766.626198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626766.626200:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626766.626202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626766.626205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626766.626215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626766.626220:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626766/real 1778626766] req@ffff95435878b800 x1865013828431616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626766.626227:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828431616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626766.626232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626766.626234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626766.626240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626766.817760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626766.817828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626767.585785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626767.841759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626767.841818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778626768.609763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626768.865764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626768.865877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626769.633856:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626769.889762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626769.889884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626770.657763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626770.913783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626770.913842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626771.681832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626771.745795:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626771.745806:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626771.745811:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626771.745821:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11106 00000100:00080000:3.0:1778626771.745835:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626771.745841:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626771.745882:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828431744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626771.745936:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626771.745941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626771.745943:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626771.745948:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11106 00000100:00080000:3.0:1778626771.745954:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626771.745958:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626771.745968:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626771.745982:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828431744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626771.745982:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828431872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626771.745997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626771.745999:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626771.746001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626771.746004:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11106 00000100:00080000:3.0:1778626771.746007:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626771.746009:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626771.746012:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626771.746020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828432000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626771.746027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626771.746030:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626771.746031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626771.746033:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11106 00000100:00080000:3.0:1778626771.746044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626771.746048:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626771.746052:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626771.746062:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828432128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626771.746069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626771.746072:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626771.746073:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626771.746076:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11106 00000100:00080000:3.0:1778626771.746078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626771.746080:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626771.746083:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626771.746091:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828432256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626771.746118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626771.746145:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626771.746161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626771.746175:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626771.937838:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626771.937971:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626772.705746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626772.961767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626772.961865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626773.729732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626773.985784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626773.985959:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626774.753787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778626774.818135:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778626774.818200:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778626774.818227:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626774.818325:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626774.818344:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626774.818360:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626774.818375:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626774.818418:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626771/real 1778626774] req@ffff95434701e680 x1865013828431872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626774.818475:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828431872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626774.818496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626774.818506:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626774.818515:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828431872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626774.818529:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626771/real 1778626774] req@ffff95434701f480 x1865013828432000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626774.818555:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828432000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626774.818571:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626774.818604:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626774.818611:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626774.818625:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626771/real 1778626774] req@ffff95434701ca80 x1865013828432128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626774.818667:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828432128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626774.818740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626774.818748:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626774.818755:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626774.818768:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626771/real 1778626774] req@ffff95434701df80 x1865013828432256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626774.818794:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828432256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626774.818811:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626774.818830:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626774.818841:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626774.818855:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626771/real 1778626774] req@ffff95434701d180 x1865013828431744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626774.818881:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828431744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626774.818907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626774.818928:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626774.818935:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828431744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626775.009733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626775.009811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626775.777777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626776.033804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626776.033972:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626776.801831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626776.865823:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626776.865835:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626776.865841:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626776.865854:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11111 00000100:00080000:3.0:1778626776.865870:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626776.865880:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626776.865928:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828432384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626776.865966:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626776.865990:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626776.865994:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626776.866001:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11111 00000100:00080000:3.0:1778626776.866008:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626776.866014:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626776.866026:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626776.866031:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828432384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626776.866047:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828432512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626776.866074:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626776.866079:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626776.866081:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626776.866086:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11111 00000100:00080000:3.0:1778626776.866091:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626776.866095:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626776.866101:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626776.866114:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828432640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626776.866138:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626776.866144:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626776.866150:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778626776.866152:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626776.866161:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11111 00000100:00080000:3.0:1778626776.866170:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626776.866180:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626776.866204:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626776.866258:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626776/real 1778626776] req@ffff95434701f800 x1865013828432384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626776.866271:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828432768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626776.866328:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626776.866340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828432384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626776.866340:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626776.866345:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626776.866360:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11111 00000100:00080000:3.0:1778626776.866402:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626776.866408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778626776.866414:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626776.866424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:3.0:1778626776.866429:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626776.866436:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828432384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626776.866458:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828432896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626776.866482:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626776.866531:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626776.866558:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626776/real 1778626776] req@ffff95434701d880 x1865013828432512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626776.866609:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828432512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626776.866624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626776.866629:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626776.866634:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626776.866644:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626776.866737:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626776.866761:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626776/real 1778626776] req@ffff95434701ea00 x1865013828432640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626776.866782:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828432640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626776.866795:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626776.866801:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626776.866805:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626776.866815:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626776.866852:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626776.866876:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626776/real 1778626776] req@ffff95434701c000 x1865013828432768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626776.866896:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828432768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626776.866910:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626776.866916:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626776.866921:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626776.866955:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626776.866987:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626776.867004:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626776/real 1778626776] req@ffff95434701d500 x1865013828432896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626776.867051:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828432896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626776.867065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626776.867069:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626776.867074:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828432896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626777.057776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626777.057897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626777.825749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626778.081806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626778.081958:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626778.849786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626779.105783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626779.105841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626779.873751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626780.129831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626780.129894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626780.897751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626781.153807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626781.153874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626781.921743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626781.985817:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626781.985833:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626781.985841:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626781.985856:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11116 00000100:00080000:0.0:1778626781.985874:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626781.985884:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626781.985935:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828433024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626781.985971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626781.985975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626781.985977:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626781.985981:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11116 00000100:00080000:0.0:1778626781.985985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626781.985988:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626781.985999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626781.986010:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828433152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626781.986050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626781.986063:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828433024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626781.986068:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626781.986076:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626781.986085:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11116 00000100:00080000:0.0:1778626781.986092:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626781.986099:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626781.986108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626781.986145:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828433280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626781.986155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626781.986159:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626781.986160:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626781.986171:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11116 00000800:00000100:2.0:1778626781.986183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778626781.986185:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626781.986197:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626781.986205:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626781.986223:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828433408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626781.986238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626781.986245:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626781/real 1778626781] req@ffff954342f48700 x1865013828433024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626781.986252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626781.986262:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626781.986277:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11116 00000100:00100000:2.0:1778626781.986286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828433024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626781.986291:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626781.986304:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778626781.986307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778626781.986312:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626781.986317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626781.986327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828433024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626781.986338:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828433536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626781.986363:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626781.986395:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626781.986405:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626781/real 1778626781] req@ffff954342f4a680 x1865013828433152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626781.986418:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828433152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626781.986455:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626781.986460:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626781.986465:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626781.986474:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626781.986503:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626781.986525:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626781/real 1778626781] req@ffff954342f49f80 x1865013828433280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626781.986536:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828433280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626781.986542:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626781.986545:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626781.986548:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626781.986553:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626781.986571:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626781.986578:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626781/real 1778626781] req@ffff954342f4b100 x1865013828433408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626781.986587:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828433408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626781.986593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626781.986595:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626781.986604:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626781.986610:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626781.986627:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626781.986635:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626781/real 1778626781] req@ffff954342f48a80 x1865013828433536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626781.986644:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828433536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626781.986652:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626781.986654:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626781.986657:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626782.177758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626782.177985:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626782.945788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626783.201822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626783.201960:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626783.969775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626784.225736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626784.225825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626784.993755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626785.249827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626785.249959:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626786.017784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626786.273775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626786.273891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626787.041746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626787.105808:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626787.105819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626787.105824:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626787.105835:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11121 00000100:00080000:3.0:1778626787.105849:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626787.105856:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626787.105900:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828433664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626787.105931:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626787.105934:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626787.105935:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626787.105938:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11121 00000100:00080000:3.0:1778626787.105961:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626787.105970:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626787.105980:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626787.105993:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828433792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626787.106004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626787.106008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626787.106010:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626787.106027:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11121 00000100:00080000:3.0:1778626787.106032:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626787.106036:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828433664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626787.106039:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626787.106048:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626787.106063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828433920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626787.106075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626787.106078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626787.106079:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626787.106082:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11121 00000100:00080000:3.0:1778626787.106097:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626787.106100:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626787.106105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626787.106114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626787.106118:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828434048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626787.106139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626787.106142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626787.106143:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626787.106148:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11121 00000100:00080000:3.0:1778626787.106151:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626787.106156:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626787.106157:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626787/real 1778626787] req@ffff95434701c700 x1865013828433664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778626787.106160:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626787.106173:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828434176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626787.106207:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828433664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626787.106217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626787.106221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626787.106224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828433664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626787.106230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626787.106251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626787.106260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626787/real 1778626787] req@ffff95434701f100 x1865013828433792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626787.106268:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828433792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626787.106274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626787.106276:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626787.106279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626787.106301:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626787.106311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626787.106317:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626787/real 1778626787] req@ffff95434701c380 x1865013828433920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626787.106324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828433920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626787.106351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626787.106355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626787.106358:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828433920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626787.106364:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626787.106388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626787.106399:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626787/real 1778626787] req@ffff95434701d500 x1865013828434048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626787.106410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828434048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626787.106416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626787.106419:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626787.106421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626787.106425:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626787.106439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626787.106446:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626787/real 1778626787] req@ffff95434701c000 x1865013828434176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626787.106453:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828434176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626787.106458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626787.106460:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626787.106467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626787.297783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626787.297867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626788.065789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626788.321763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626788.321911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626789.089767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626789.345765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626789.345844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626790.113777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626790.369739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626790.369802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626791.137746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626791.393731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626791.393816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626792.161725:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626792.225787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626792.225795:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626792.225799:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626792.225808:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11126 00000100:00080000:3.0:1778626792.225818:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626792.225824:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626792.225859:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828434304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626792.225885:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626792.225888:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626792.225890:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626792.225893:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11126 00000100:00080000:3.0:1778626792.225896:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626792.225898:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626792.225938:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626792.225955:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828434432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626792.225970:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626792.225974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778626792.225979:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828434304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626792.225979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626792.225987:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11126 00000100:00080000:3.0:1778626792.225992:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626792.225995:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626792.226000:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626792.226011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828434560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626792.226019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626792.226022:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626792.226023:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626792.226026:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11126 00000100:00080000:3.0:1778626792.226029:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626792.226032:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626792.226036:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626792.226043:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626792.226046:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828434688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626792.226054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626792.226057:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626792.226064:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626792.226068:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11126 00000100:00080000:3.0:1778626792.226072:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626792.226075:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626792.226080:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626792.226082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626792/real 1778626792] req@ffff95434701ea00 x1865013828434304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626792.226092:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828434816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626792.226102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828434304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626792.226111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626792.226115:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626792.226119:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828434304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626792.226138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626792.226153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626792.226161:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626792/real 1778626792] req@ffff95434701d880 x1865013828434432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626792.226170:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828434432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626792.226184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626792.226186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626792.226189:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626792.226193:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626792.226207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626792.226214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626792/real 1778626792] req@ffff95434701f800 x1865013828434560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626792.226222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828434560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626792.226227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626792.226229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626792.226231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626792.226236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626792.226247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626792.226252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626792/real 1778626792] req@ffff95434701d180 x1865013828434688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626792.226277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828434688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626792.226282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626792.226283:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626792.226290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626792.226294:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626792.226306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626792.226312:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626792/real 1778626792] req@ffff95434701df80 x1865013828434816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626792.226320:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828434816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626792.226325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626792.226327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626792.226329:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828434816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626792.417730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626792.417797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626793.185749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626793.441767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626793.441847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626794.209828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626794.465773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626794.465838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626795.233751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626795.489757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626795.489837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626796.257848:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626796.513756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626796.513855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626797.281736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626797.345831:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626797.345839:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626797.345843:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626797.345851:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11131 00000100:00080000:2.0:1778626797.345860:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626797.345881:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626797.345909:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828434944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626797.345938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626797.345942:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626797.345944:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626797.345948:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11131 00000100:00080000:2.0:1778626797.345953:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626797.345957:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626797.345966:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626797.345981:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828435072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626797.345992:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828434944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626797.345996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626797.346000:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626797.346001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626797.346012:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11131 00000100:00080000:2.0:1778626797.346015:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626797.346019:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626797.346023:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626797.346033:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828435200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626797.346040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626797.346042:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626797.346048:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626797.346052:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11131 00000100:00080000:2.0:1778626797.346073:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778626797.346076:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626797.346080:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626797.346086:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626797.346100:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828435328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626797.346109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626797.346112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626797.346113:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626797.346116:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11131 00000100:00080000:2.0:1778626797.346135:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626797.346141:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626797.346142:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626797/real 1778626797] req@ffff954358789c00 x1865013828434944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778626797.346147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626797.346160:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828435456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626797.346168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828434944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626797.346178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626797.346181:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626797.346185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828434944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626797.346207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626797.346225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626797.346233:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626797/real 1778626797] req@ffff954358789180 x1865013828435072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626797.346242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828435072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626797.346248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626797.346251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626797.346260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626797.346265:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626797.346280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626797.346287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626797/real 1778626797] req@ffff954358788e00 x1865013828435200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626797.346296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828435200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626797.346301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626797.346303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626797.346305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626797.346310:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626797.346333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626797.346339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626797/real 1778626797] req@ffff954358789500 x1865013828435328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626797.346346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828435328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626797.346352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626797.346354:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626797.346356:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626797.346360:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626797.346370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626797.346383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626797/real 1778626797] req@ffff954358788a80 x1865013828435456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626797.346399:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828435456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626797.346414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626797.346416:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626797.346418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626797.537833:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626797.537899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626798.305847:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626798.561818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626798.561911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626799.329846:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626799.585849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626799.585876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626800.353807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626800.609805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626800.609875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626801.377760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626801.633830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626801.633904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626802.401820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626802.465862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626802.465872:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626802.465877:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626802.465887:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11136 00000100:00080000:1.0:1778626802.465898:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626802.465906:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626802.465944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828435584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626802.465972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626802.465976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626802.465978:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626802.466000:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11136 00000100:00080000:1.0:1778626802.466004:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626802.466009:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626802.466019:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626802.466039:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828435712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626802.466049:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828435584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626802.466065:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626802.466070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626802.466073:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626802.466081:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11136 00000100:00080000:1.0:1778626802.466087:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626802.466093:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626802.466101:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626802.466121:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828435840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626802.466130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626802.466147:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626802.466152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626802.466156:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626802.466164:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11136 00000100:00080000:1.0:1778626802.466171:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626802.466176:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626802.466184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626802.466187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626802/real 1778626802] req@ffff954350195f80 x1865013828435584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626802.466200:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828435968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626802.466219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626802.466223:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828435584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626802.466224:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626802.466229:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626802.466241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626802.466247:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778626802.466253:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11136 00000100:00100000:0.0:1778626802.466254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828435584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626802.466262:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626802.466269:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626802.466275:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626802.466286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778626802.466293:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828436096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626802.466321:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626802.466338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626802/real 1778626802] req@ffff954350195180 x1865013828435712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626802.466350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828435712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626802.466374:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626802.466377:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626802.466381:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626802.466387:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626802.466406:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626802.466415:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626802/real 1778626802] req@ffff954350197b80 x1865013828435840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626802.466426:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828435840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626802.466433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626802.466435:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626802.466438:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626802.466443:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626802.466476:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626802.466485:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626802/real 1778626802] req@ffff954350195880 x1865013828435968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626802.466496:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828435968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626802.466504:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626802.466506:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626802.466516:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828435968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626802.466532:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626802.466551:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626802.466560:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626802/real 1778626802] req@ffff954350194e00 x1865013828436096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626802.466571:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828436096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626802.466580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626802.466600:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626802.466602:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626802.657884:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626802.657971:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626803.425788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626803.681835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626803.682041:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626804.449813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626804.705841:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626804.706090:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626805.473756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626805.730173:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626805.730534:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626806.497730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626806.753780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626806.753851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626807.521753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626807.585833:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626807.585841:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626807.585845:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626807.585853:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11142 00000100:00080000:3.0:1778626807.585862:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626807.585890:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626807.585951:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828436224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626807.585975:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626807.585978:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626807.585980:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626807.585983:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11142 00000100:00080000:3.0:1778626807.585986:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626807.585989:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626807.585996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626807.586021:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828436352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626807.586047:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626807.586050:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828436224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626807.586051:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626807.586054:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626807.586062:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11142 00000100:00080000:3.0:1778626807.586065:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626807.586067:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626807.586071:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626807.586080:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828436480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626807.586087:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626807.586090:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626807.586091:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626807.586093:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11142 00000100:00080000:3.0:1778626807.586096:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626807.586098:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626807.586101:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626807.586109:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828436608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626807.586118:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626807.586119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626807.586122:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626807.586125:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626807.586133:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11142 00000100:00080000:3.0:1778626807.586137:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626807.586141:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626807.586145:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626807.586153:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828436736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626807.586164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626807/real 1778626807] req@ffff95434701ca80 x1865013828436224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626807.586194:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828436224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626807.586206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626807.586211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626807.586215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828436224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626807.586237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626807.586264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626807.586287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626807/real 1778626807] req@ffff95434701f480 x1865013828436352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626807.586302:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828436352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626807.586311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626807.586315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626807.586318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626807.586324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626807.586343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626807.586353:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626807/real 1778626807] req@ffff95434701e680 x1865013828436480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626807.586365:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828436480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626807.586373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626807.586376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626807.586380:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626807.586386:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626807.586405:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626807.586416:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626807/real 1778626807] req@ffff95434701df80 x1865013828436608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626807.586429:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828436608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626807.586446:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626807.586449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626807.586453:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626807.586459:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626807.586480:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626807.586490:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626807/real 1778626807] req@ffff95434701d180 x1865013828436736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626807.586502:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828436736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626807.586510:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626807.586514:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626807.586517:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626807.777790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626807.777813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626808.545771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626808.801771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626808.801853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626809.569810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626809.825729:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626809.825765:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626810.593740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626810.849736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626810.849785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626811.617745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626811.873734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626811.873772:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626812.641777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626812.705812:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626812.705821:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626812.705825:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626812.705834:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11147 00000100:00080000:3.0:1778626812.705845:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626812.705852:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626812.705887:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828436864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626812.705915:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626812.705918:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626812.705920:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626812.705924:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11147 00000100:00080000:3.0:1778626812.705927:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626812.705929:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626812.705956:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626812.705968:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828436992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626812.705977:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828436864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626812.705982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626812.705987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626812.705988:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626812.705992:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11147 00000100:00080000:3.0:1778626812.705996:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626812.705999:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626812.706004:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626812.706016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828437120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626812.706024:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626812.706027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626812.706028:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626812.706032:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11147 00000100:00080000:3.0:1778626812.706035:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626812.706038:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626812.706042:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626812.706046:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626812.706057:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828437248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626812.706066:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626812.706069:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626812.706075:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626812.706079:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11147 00000100:00080000:3.0:1778626812.706083:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626812.706084:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626812/real 1778626812] req@ffff95434701f800 x1865013828436864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626812.706093:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626812.706106:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626812.706111:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828436864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626812.706129:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828437376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626812.706134:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626812.706143:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626812.706151:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828436864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626812.706178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626812.706201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626812.706211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626812/real 1778626812] req@ffff95434701d880 x1865013828436992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626812.706221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828436992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626812.706228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626812.706231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626812.706233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828436992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626812.706238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626812.706250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626812.706256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626812/real 1778626812] req@ffff95434701ea00 x1865013828437120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626812.706265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828437120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626812.706271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626812.706273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626812.706275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626812.706286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626812.706302:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626812.706310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626812/real 1778626812] req@ffff95434701c000 x1865013828437248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626812.706318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828437248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626812.706325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626812.706327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626812.706330:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626812.706334:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626812.706345:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626812.706351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626812/real 1778626812] req@ffff95434701d500 x1865013828437376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626812.706360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828437376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626812.706380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626812.706384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626812.706388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626812.897770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626812.897799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626813.665813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626813.921766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626813.921818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626814.689737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626814.945752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626814.945792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626815.713818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626815.969771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626815.969787:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626816.737733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626816.993752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626816.993762:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626817.761775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626817.825742:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626817.825751:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626817.825755:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626817.825765:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11152 00000100:00080000:0.0:1778626817.825777:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626817.825784:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626817.825821:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828437504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626817.825851:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626817.825854:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626817.825856:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626817.825859:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11152 00000100:00080000:0.0:1778626817.825862:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626817.825865:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626817.825872:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626817.825883:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828437632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626817.825891:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626817.825893:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626817.825895:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626817.825897:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11152 00000100:00080000:0.0:1778626817.825900:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626817.825903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828437504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626817.825904:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626817.825920:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626817.825931:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828437760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626817.825939:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626817.825949:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626817.825950:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626817.825953:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11152 00000100:00080000:0.0:1778626817.825957:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626817.825959:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626817.825963:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626817.825969:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626817.825973:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828437888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626817.825980:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626817.825983:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626817.825984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626817.825987:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11152 00000100:00080000:0.0:1778626817.825990:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626817.825993:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626817.825996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626817.826007:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828438016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626817.826008:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626817/real 1778626817] req@ffff954342f4ad80 x1865013828437504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626817.826041:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828437504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626817.826052:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626817.826066:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626817.826070:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828437504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626817.826079:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626817.826118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626817.826133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626817/real 1778626817] req@ffff954342f49500 x1865013828437632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626817.826146:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828437632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626817.826155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626817.826158:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626817.826162:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626817.826185:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626817.826204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626817.826213:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626817/real 1778626817] req@ffff954342f49c00 x1865013828437760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626817.826226:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828437760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626817.826234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626817.826238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626817.826241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626817.826247:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626817.826274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626817.826285:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626817/real 1778626817] req@ffff954342f4b480 x1865013828437888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626817.826297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828437888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626817.826306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626817.826309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626817.826312:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828437888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626817.826319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626817.826338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626817.826348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626817/real 1778626817] req@ffff954342f4a300 x1865013828438016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626817.826360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828438016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626817.826369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626817.826372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626817.826376:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626818.017778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626818.017801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626818.785775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626819.041800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626819.041868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626819.809833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626820.065790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626820.065863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626820.833740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626821.089787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626821.089850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626821.857814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626822.113818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626822.113886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626822.881763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626822.945784:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626822.945814:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626822.945818:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626822.945829:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11157 00000100:00080000:0.0:1778626822.945843:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626822.945850:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626822.945889:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828438144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626822.945917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626822.945920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626822.945922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626822.945925:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11157 00000100:00080000:0.0:1778626822.945928:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626822.945930:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626822.945939:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626822.945949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828438272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626822.945957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626822.945960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626822.945962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626822.945966:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828438144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626822.945967:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11157 00000100:00080000:0.0:1778626822.945972:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626822.945977:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626822.945982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626822.946002:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828438400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626822.946020:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626822.946024:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626822.946026:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626822.946031:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11157 00000100:00080000:0.0:1778626822.946037:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626822.946042:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778626822.946047:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778626822.946050:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626822.946066:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828438528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626822.946074:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626822.946078:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626822.946080:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626822.946085:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626822/real 1778626822] req@ffff954342f49880 x1865013828438144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626822.946086:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11157 00000100:00080000:0.0:1778626822.946094:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626822.946100:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626822.946107:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626822.946109:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828438144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626822.946120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778626822.946123:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828438656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626822.946126:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626822.946132:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828438144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626822.946163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626822.946181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626822.946189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626822/real 1778626822] req@ffff954342f4bb80 x1865013828438272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626822.946198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828438272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626822.946204:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626822.946206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626822.946208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626822.946220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626822.946234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626822.946240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626822/real 1778626822] req@ffff954342f49180 x1865013828438400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626822.946248:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828438400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626822.946253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626822.946256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626822.946258:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626822.946261:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626822.946272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626822.946277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626822/real 1778626822] req@ffff954342f48000 x1865013828438528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626822.946284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828438528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626822.946289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626822.946291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626822.946293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626822.946296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626822.946317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626822.946324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626822/real 1778626822] req@ffff954342f48380 x1865013828438656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626822.946338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828438656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626822.946344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626822.946346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626822.946349:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626823.137771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626823.137828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626823.905807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626824.161798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626824.161876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626824.929763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626825.185757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626825.185826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626825.953810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626826.209780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626826.209835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626826.977801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626827.233812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626827.233827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626828.001833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626828.065820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626828.065831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626828.065837:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626828.065850:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11162 00000100:00080000:3.0:1778626828.065866:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626828.065875:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626828.065927:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828438784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626828.065954:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626828.065957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626828.065977:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626828.065984:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828438784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626828.065984:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11162 00000100:00080000:3.0:1778626828.065990:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626828.065995:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626828.066006:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626828.066016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828438912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626828.066025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626828.066028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626828.066029:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626828.066033:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11162 00000100:00080000:3.0:1778626828.066035:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626828.066038:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626828.066041:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626828.066050:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828439040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626828.066056:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626828.066058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626828.066059:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626828.066062:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11162 00000100:00080000:3.0:1778626828.066065:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626828.066068:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626828.066070:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626828.066085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626828.066087:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828439168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626828.066095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626828.066098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626828.066099:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626828.066102:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11162 00000100:00080000:3.0:1778626828.066106:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626828.066109:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626828.066112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626828.066119:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828439296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626828.066141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626828/real 1778626828] req@ffff95434701c380 x1865013828438784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626828.066166:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828438784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626828.066179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626828.066189:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626828.066192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828438784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626828.066237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626828.066258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626828.066268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626828/real 1778626828] req@ffff95434701f100 x1865013828438912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626828.066277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828438912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626828.066284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626828.066287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626828.066289:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828438912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626828.066293:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626828.066305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626828.066310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626828/real 1778626828] req@ffff95434701c700 x1865013828439040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626828.066318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828439040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626828.066323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626828.066326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626828.066328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626828.066331:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626828.066342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626828.066347:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626828/real 1778626828] req@ffff95434701d500 x1865013828439168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626828.066360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828439168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626828.066366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626828.066369:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626828.066371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626828.066374:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626828.066387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626828.066392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626828/real 1778626828] req@ffff95434701c000 x1865013828439296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626828.066400:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828439296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626828.066405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626828.066407:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626828.066409:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626828.257804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626828.257870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626829.025807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626829.281781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626829.281854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626830.049758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626830.305774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626830.305859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626831.073818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626831.329791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626831.329893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626832.097798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626832.353832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626832.353937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626833.121767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626833.185775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626833.185785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626833.185790:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626833.185799:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11167 00000100:00080000:2.0:1778626833.185810:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626833.185817:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626833.185856:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828439424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626833.185887:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626833.185891:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626833.185893:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626833.185898:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11167 00000100:00080000:2.0:1778626833.185902:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626833.185905:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626833.185925:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626833.185941:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828439552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626833.185953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626833.185959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828439424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626833.185960:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626833.185963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626833.185976:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11167 00000100:00080000:2.0:1778626833.185981:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626833.185985:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626833.186019:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626833.186036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828439680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626833.186047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626833.186051:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626833.186052:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626833.186057:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11167 00000100:00080000:2.0:1778626833.186063:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626833.186067:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626833.186073:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626833.186081:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626833.186088:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828439808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626833.186117:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626833.186122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626833.186125:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626833.186129:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11167 00000100:00080000:2.0:1778626833.186136:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626833.186142:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626833.186148:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626833.186150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626833/real 1778626833] req@ffff954358789880 x1865013828439424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626833.186182:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828439936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626833.186212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828439424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626833.186222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626833.186227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626833.186230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828439424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626833.186262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626833.186290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626833.186302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626833/real 1778626833] req@ffff954358788700 x1865013828439552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626833.186311:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828439552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626833.186318:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626833.186320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626833.186323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626833.186328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626833.186342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626833.186348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626833/real 1778626833] req@ffff95435878aa00 x1865013828439680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626833.186356:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828439680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626833.186371:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626833.186374:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626833.186376:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626833.186381:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626833.186397:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626833.186403:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626833/real 1778626833] req@ffff95435878b480 x1865013828439808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626833.186411:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828439808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626833.186416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626833.186418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626833.186420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626833.186424:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626833.186434:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626833.186440:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626833/real 1778626833] req@ffff954358788380 x1865013828439936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626833.186447:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828439936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626833.186453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626833.186455:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626833.186464:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828439936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626833.377821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626833.377929:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626834.145788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626834.401807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626834.401893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626835.169805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626835.425808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626835.425890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626836.193748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626836.449765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626836.449962:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626837.217745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626837.473765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626837.473834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626838.241733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626838.305753:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626838.305762:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626838.305765:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626838.305773:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11172 00000100:00080000:2.0:1778626838.305782:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626838.305787:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626838.305818:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828440064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626838.305843:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626838.305846:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626838.305847:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626838.305850:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11172 00000100:00080000:2.0:1778626838.305852:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626838.305854:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626838.305861:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626838.305869:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828440192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626838.305887:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626838.305903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626838.305905:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626838.305910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11172 00000100:00100000:0.0:1778626838.305912:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828440064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626838.305913:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626838.305919:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626838.305925:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626838.305935:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828440320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626838.305953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626838.305957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626838.305959:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626838.305963:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11172 00000100:00080000:2.0:1778626838.305968:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626838.305971:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626838.305976:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626838.305989:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828440448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626838.306012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626838.306017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626838.306019:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626838.306025:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11172 00000100:00100000:0.0:1778626838.306030:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440192:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778626838.306030:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626838.306036:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626838.306046:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626838.306060:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828440576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626838.306071:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626838.306092:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626838.306107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626838.497743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626838.497801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626839.265773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626839.521770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626839.521837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626840.289759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626840.545742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626840.545787:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626841.313818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778626841.377917:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778626841.377982:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778626841.378001:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626841.378042:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626841.378050:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626841.378056:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778626841.378063:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626841.378093:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626838/real 1778626841] req@ffff95435878aa00 x1865013828440576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626841.378130:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828440576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626841.378145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626841.378155:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626841.378160:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626841.378169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626838/real 1778626841] req@ffff954358788380 x1865013828440320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626841.378183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828440320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626841.378191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626841.378196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626841.378199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626841.378229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626838/real 1778626841] req@ffff95435878b480 x1865013828440448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626841.378242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828440448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626841.378250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626841.378254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626841.378263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626841.378270:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626838/real 1778626841] req@ffff95435878bb80 x1865013828440064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626841.378284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828440064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626841.378291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626841.378295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626841.378298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828440064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626841.378306:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626838/real 1778626841] req@ffff95435878b100 x1865013828440192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626841.378323:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828440192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626841.378332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626841.378336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626841.378339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626841.569798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626841.569863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626842.337792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626842.593789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626842.593844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626843.361782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626843.425830:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626843.425840:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626843.425848:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626843.425859:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11177 00000100:00080000:3.0:1778626843.425882:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626843.425889:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626843.425934:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828440704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626843.425979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626843.425983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626843.425985:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626843.425990:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11177 00000100:00080000:3.0:1778626843.425994:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626843.425998:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626843.426008:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626843.426047:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828440704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626843.426051:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828440832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626843.426064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626843.426068:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626843.426070:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626843.426074:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11177 00000100:00080000:3.0:1778626843.426078:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626843.426083:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626843.426088:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626843.426100:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828440960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626843.426111:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626843.426116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626843.426123:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626843.426126:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626843.426131:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11177 00000100:00080000:3.0:1778626843.426137:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626843.426142:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626843.426148:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626843.426155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626843/real 1778626843] req@ffff95434701ea00 x1865013828440704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626843.426165:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828441088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626843.426177:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626843.426181:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778626843.426183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828440704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626843.426184:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626843.426193:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11177 00000100:00080000:3.0:1778626843.426200:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626843.426206:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778626843.426207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1778626843.426211:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626843.426215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626843.426220:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828440704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626843.426235:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828441216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626843.426252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626843.426279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626843.426295:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626843/real 1778626843] req@ffff95434701d880 x1865013828440832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626843.426318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828440832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626843.426326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626843.426330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626843.426333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626843.426342:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626843.426364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626843.426375:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626843/real 1778626843] req@ffff95434701f800 x1865013828440960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626843.426386:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828440960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626843.426392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626843.426395:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626843.426398:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828440960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626843.426403:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626843.426421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626843.426430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626843/real 1778626843] req@ffff95434701d180 x1865013828441088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626843.426439:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828441088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626843.426459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626843.426463:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626843.426466:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626843.426485:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626843.426504:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626843.426515:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626843/real 1778626843] req@ffff95434701df80 x1865013828441216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626843.426528:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828441216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626843.426544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626843.426547:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626843.426550:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626843.617756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626843.617816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626844.385755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626844.641771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626844.641827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626845.409791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626845.665772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626845.665823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626846.433754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626846.689743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626846.689821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626847.457784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626847.713786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626847.713854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626848.481770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626848.545733:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626848.545740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626848.545744:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626848.545754:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11183 00000100:00080000:3.0:1778626848.545765:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626848.545771:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626848.545820:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828441344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626848.545859:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626848.545863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626848.545865:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626848.545869:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11183 00000100:00080000:3.0:1778626848.545872:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626848.545875:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626848.545883:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626848.545892:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828441472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626848.545899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626848.545902:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626848.545904:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626848.545906:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11183 00000100:00080000:3.0:1778626848.545908:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626848.545913:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828441344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626848.545913:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626848.545916:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626848.545925:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828441600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626848.545940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626848.545943:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626848.545945:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626848.545949:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11183 00000100:00080000:3.0:1778626848.545971:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626848.545976:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778626848.545982:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778626848.545983:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626848.545998:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828441728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626848.546006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626848.546008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626848.546010:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626848.546013:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11183 00000100:00080000:3.0:1778626848.546017:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626848.546023:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626848/real 1778626848] req@ffff95434701e680 x1865013828441344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626848.546023:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626848.546033:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626848.546043:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828441344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626848.546048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828441856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626848.546058:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626848.546066:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626848.546070:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828441344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626848.546076:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626848.546091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626848.546106:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626848/real 1778626848] req@ffff95434701f480 x1865013828441472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626848.546120:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828441472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626848.546129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626848.546133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626848.546136:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626848.546160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626848.546176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626848.546184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626848/real 1778626848] req@ffff95434701ca80 x1865013828441600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626848.546202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828441600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626848.546207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626848.546209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626848.546211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626848.546215:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626848.546228:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626848.546234:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626848/real 1778626848] req@ffff95434701df80 x1865013828441728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626848.546241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828441728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626848.546246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626848.546248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626848.546249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626848.546253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626848.546262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626848.546266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626848/real 1778626848] req@ffff95434701d180 x1865013828441856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626848.546273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828441856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626848.546287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626848.546289:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626848.546291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828441856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626848.737742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626848.737818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626849.505767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626849.761749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626849.761802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626850.529787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626850.785768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626850.785953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626851.553802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626851.809801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626851.809862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626852.577827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626852.833803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626852.833871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626853.601782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626853.665780:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626853.665788:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626853.665792:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626853.665801:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11188 00000100:00080000:3.0:1778626853.665811:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626853.665816:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626853.665848:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828441984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626853.665870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626853.665885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626853.665887:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626853.665890:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11188 00000100:00080000:3.0:1778626853.665894:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626853.665897:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626853.665905:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626853.665915:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828442112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626853.665920:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828441984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626853.665924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626853.665930:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626853.665931:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626853.665934:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11188 00000100:00080000:3.0:1778626853.665936:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626853.665938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626853.665941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626853.665948:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828442240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626853.665954:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626853.665956:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626853.665958:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626853.665960:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11188 00000100:00080000:3.0:1778626853.665974:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626853.665977:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626853.665987:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778626853.665991:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626853.665999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828442368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626853.666007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626853.666009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626853.666010:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626853.666013:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11188 00000100:00080000:3.0:1778626853.666015:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626853.666018:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626853.666021:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626853.666031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828442496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626853.666036:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626853/real 1778626853] req@ffff95434701f800 x1865013828441984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626853.666075:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828441984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626853.666084:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626853.666087:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626853.666090:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828441984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626853.666110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626853.666128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626853.666145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626853/real 1778626853] req@ffff95434701d880 x1865013828442112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626853.666154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828442112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626853.666160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626853.666162:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626853.666165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626853.666169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626853.666181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626853.666187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626853/real 1778626853] req@ffff95434701ea00 x1865013828442240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626853.666195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828442240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626853.666200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626853.666203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626853.666205:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626853.666208:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626853.666218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626853.666224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626853/real 1778626853] req@ffff95434701c000 x1865013828442368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626853.666231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828442368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626853.666241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626853.666244:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626853.666246:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626853.666250:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626853.666262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626853.666268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626853/real 1778626853] req@ffff95434701d500 x1865013828442496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626853.666275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828442496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626853.666281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626853.666283:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626853.666285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626853.857785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626853.857881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626854.625760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626854.881792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626854.881848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626855.649824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626855.905760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626855.905838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626856.673918:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626856.929777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626856.929857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626857.697792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626857.953784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626857.953862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626858.721792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626858.785794:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626858.785802:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626858.785821:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626858.785831:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11193 00000100:00080000:2.0:1778626858.785841:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626858.785847:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626858.785876:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828442624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626858.785902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626858.785905:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626858.785906:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626858.785910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11193 00000100:00080000:2.0:1778626858.785913:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626858.785916:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626858.785924:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626858.785932:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828442752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626858.785939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626858.785942:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626858.785950:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626858.785953:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11193 00000100:00080000:2.0:1778626858.785955:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626858.785959:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626858.785963:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626858.785971:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828442880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626858.785983:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828442624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626858.786010:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626858.786012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626858.786017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626858.786023:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11193 00000100:00080000:2.0:1778626858.786026:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626858.786029:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626858.786034:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626858.786044:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828443008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626858.786054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626858.786057:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626858.786059:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626858.786063:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11193 00000100:00080000:2.0:1778626858.786068:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626858.786073:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626858.786078:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626858.786084:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626858.786096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828443136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626858.786139:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626858/real 1778626858] req@ffff954358788700 x1865013828442624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626858.786158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828442624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626858.786167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626858.786171:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626858.786174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828442624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626858.786180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626858.786198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626858.786206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626858/real 1778626858] req@ffff954358789880 x1865013828442752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626858.786214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828442752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626858.786220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626858.786223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626858.786225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626858.786248:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626858.786259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626858.786265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626858/real 1778626858] req@ffff954358788a80 x1865013828442880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626858.786286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828442880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626858.786291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626858.786294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626858.786296:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828442880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626858.786300:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626858.786313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626858.786319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626858/real 1778626858] req@ffff954358789500 x1865013828443008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626858.786326:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828443008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626858.786332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626858.786334:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626858.786335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626858.786339:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626858.786348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626858.786353:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626858/real 1778626858] req@ffff954358788e00 x1865013828443136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626858.786361:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828443136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626858.786376:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626858.786378:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626858.786380:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626858.977769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626858.977840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626859.745738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626860.001779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626860.001799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626860.769835:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626861.025782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626861.025839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626861.793731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626862.049744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626862.049815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626862.817789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626863.073794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626863.073853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626863.841763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626863.905776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626863.905785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626863.905788:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626863.905797:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11198 00000100:00080000:3.0:1778626863.905813:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626863.905819:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626863.905850:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828443264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626863.905871:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626863.905873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626863.905875:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626863.905878:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11198 00000100:00080000:3.0:1778626863.905894:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626863.905897:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626863.905904:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626863.905913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828443392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626863.905921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626863.905925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626863.905927:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626863.905932:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11198 00000100:00080000:3.0:1778626863.905935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626863.905939:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626863.905945:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626863.905950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828443264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626863.905959:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828443520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626863.905971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626863.905975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626863.905977:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626863.905981:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11198 00000100:00080000:3.0:1778626863.905986:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626863.905990:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626863.905995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626863.906008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828443648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626863.906009:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626863.906027:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626863.906039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778626863.906042:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626863/real 1778626863] req@ffff95434701c700 x1865013828443264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626863.906043:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626863.906050:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11198 00000100:00080000:3.0:1778626863.906057:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626863.906061:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626863.906065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626863.906066:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828443264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626863.906075:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828443776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626863.906087:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626863.906092:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626863.906095:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828443264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626863.906102:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626863.906117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626863.906125:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626863/real 1778626863] req@ffff95434701f100 x1865013828443392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626863.906133:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828443392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626863.906139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626863.906141:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626863.906143:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626863.906168:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626863.906187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626863.906198:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626863/real 1778626863] req@ffff95434701c380 x1865013828443520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626863.906211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828443520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626863.906219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626863.906223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626863.906226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626863.906232:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626863.906253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626863.906264:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626863/real 1778626863] req@ffff95434701d500 x1865013828443648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626863.906275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828443648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626863.906283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626863.906287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626863.906290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626863.906307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626863.906329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626863.906340:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626863/real 1778626863] req@ffff95434701c000 x1865013828443776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626863.906359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828443776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626863.906368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626863.906372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626863.906375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828443776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626864.097749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626864.097860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626864.865806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626865.121806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626865.121880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626865.889774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626866.145739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626866.145814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626866.913748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626867.169757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626867.169821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626867.937780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626868.193744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626868.193880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626868.961778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626869.025797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626869.025807:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626869.025811:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626869.025818:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11203 00000100:00080000:0.0:1778626869.025829:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626869.025835:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626869.025867:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828443904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626869.025894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626869.025897:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626869.025898:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626869.025901:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11203 00000100:00080000:0.0:1778626869.025903:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626869.025906:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626869.025913:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626869.025926:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828444032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626869.025930:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828443904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626869.025957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626869.025961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626869.025963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626869.025969:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11203 00000100:00080000:0.0:1778626869.025977:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626869.025982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778626869.025987:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778626869.025989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626869.026007:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828444160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626869.026019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626869.026022:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626869.026025:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626869.026028:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626869/real 1778626869] req@ffff954342f4aa00 x1865013828443904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626869.026032:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11203 00000100:00080000:0.0:1778626869.026038:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626869.026045:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626869.026050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626869.026056:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828443904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626869.026062:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828444288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626869.026071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626869.026074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626869.026077:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778626869.026078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626869.026082:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626869.026084:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828443904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626869.026088:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11203 00000100:00080000:0.0:1778626869.026103:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626869.026108:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444032:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778626869.026109:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626869.026118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626869.026130:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828444416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626869.026138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626869.026151:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626869/real 1778626869] req@ffff954342f48e00 x1865013828444032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626869.026168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828444032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626869.026174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626869.026177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626869.026179:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626869.026183:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626869.026196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626869.026203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626869/real 1778626869] req@ffff954342f48380 x1865013828444160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626869.026211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828444160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626869.026215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626869.026217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626869.026219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626869.026223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626869.026233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626869.026238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626869/real 1778626869] req@ffff954342f48000 x1865013828444288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626869.026245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828444288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626869.026263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626869.026267:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626869.026270:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626869.026288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626869.026310:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626869.026321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626869/real 1778626869] req@ffff954342f49180 x1865013828444416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626869.026329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828444416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626869.026334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626869.026337:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626869.026340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626869.217779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626869.217824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626869.985760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626870.241939:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626870.242057:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626871.009762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626871.265777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626871.266028:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626872.033788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626872.289846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626872.289890:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778626873.057753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626873.313758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626873.313973:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626874.081773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626874.145744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626874.145753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626874.145758:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626874.145768:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11208 00000100:00080000:3.0:1778626874.145779:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626874.145786:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626874.145822:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828444544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626874.145850:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626874.145854:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626874.145856:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626874.145860:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11208 00000100:00080000:3.0:1778626874.145865:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626874.145869:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626874.145891:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828444544:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778626874.145891:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626874.145905:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828444672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626874.145915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626874.145918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626874.145919:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626874.145922:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11208 00000100:00080000:3.0:1778626874.145926:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626874.145930:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626874.145935:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626874.145947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828444800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626874.145959:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778626874.145961:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626874.145966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626874.145968:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626874.145974:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11208 00000100:00080000:3.0:1778626874.145979:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626874.145983:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626874.145989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626874.145995:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626874/real 1778626874] req@ffff95434701ea00 x1865013828444544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778626874.146004:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828444928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626874.146014:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828444544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626874.146018:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626874.146027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626874.146031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626874.146035:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778626874.146039:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626874.146040:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828444544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626874.146046:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11208 00000100:00080000:3.0:1778626874.146050:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626874.146053:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626874.146058:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626874.146063:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778626874.146070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828445056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626874.146083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626874.146095:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626874/real 1778626874] req@ffff95434701d880 x1865013828444672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626874.146105:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828444672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626874.146110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626874.146112:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626874.146114:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626874.146118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626874.146127:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626874.146133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626874/real 1778626874] req@ffff95434701f800 x1865013828444800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626874.146151:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828444800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626874.146156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626874.146158:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626874.146160:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626874.146164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626874.146177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626874.146182:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626874/real 1778626874] req@ffff95434701d180 x1865013828444928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626874.146189:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828444928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626874.146194:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626874.146196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626874.146198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828444928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626874.146208:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626874.146217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626874.146222:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626874/real 1778626874] req@ffff95434701df80 x1865013828445056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626874.146230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828445056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626874.146245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626874.146248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626874.146252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445056:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778626874.337846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626874.338153:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778626875.105730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626875.361925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626875.362086:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778626876.129772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626876.385734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626876.385765:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626877.153773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626877.409732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626877.409804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626878.177760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626878.433810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626878.433874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626879.201764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626879.265791:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626879.265804:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626879.265810:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626879.265822:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11213 00000100:00080000:3.0:1778626879.265839:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626879.265846:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626879.265899:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828445184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626879.265936:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626879.265940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626879.265942:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626879.265947:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11213 00000100:00080000:3.0:1778626879.265951:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626879.265955:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626879.266003:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626879.266013:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828445184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626879.266028:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828445312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626879.266043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626879.266047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626879.266049:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626879.266055:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11213 00000100:00080000:3.0:1778626879.266073:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778626879.266079:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626879.266085:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626879.266093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626879.266113:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828445440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626879.266122:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626879/real 1778626879] req@ffff95434701ca80 x1865013828445184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626879.266146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626879.266150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626879.266153:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778626879.266158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828445184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626879.266161:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11213 00000100:00080000:3.0:1778626879.266174:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626879.266176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778626879.266186:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778626879.266187:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626879.266208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828445184:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778626879.266217:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626879.266247:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778626879.266257:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828445568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626879.266270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626879.266272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626879.266278:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778626879.266282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626879/real 1778626879] req@ffff95434701f480 x1865013828445312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626879.266292:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626879.266299:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11213 00000100:00080000:3.0:1778626879.266305:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626879.266320:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626879.266324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828445312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778626879.266335:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626879.266337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626879.266346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626879.266360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626879.266366:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778626879.266376:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828445696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626879.266388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626879.266403:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626879/real 1778626879] req@ffff95434701e680 x1865013828445440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626879.266412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828445440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626879.266417:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626879.266420:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626879.266422:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626879.266448:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626879.266461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626879.266467:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626879/real 1778626879] req@ffff95434701ca80 x1865013828445568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626879.266473:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828445568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626879.266487:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626879.266490:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626879.266492:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626879.266496:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626879.266512:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626879.266523:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626879/real 1778626879] req@ffff95434701df80 x1865013828445696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626879.266534:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828445696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626879.266544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626879.266548:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626879.266551:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626879.457758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626879.457823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626880.225762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626880.481760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626880.481827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626881.249744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626881.505749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626881.505815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626882.273799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626882.529761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626882.529787:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626883.297740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626883.553736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626883.553812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626884.321777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626884.385842:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626884.385851:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626884.385855:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626884.385863:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11218 00000100:00080000:1.0:1778626884.385872:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626884.385879:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626884.385913:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828445824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626884.385957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626884.385961:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626884.385962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626884.385965:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11218 00000100:00080000:1.0:1778626884.385968:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626884.385971:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626884.385978:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626884.385986:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828445952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626884.385992:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626884.385995:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626884.385996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626884.385998:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11218 00000100:00080000:1.0:1778626884.385999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626884.386001:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626884.386004:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626884.386009:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828446080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626884.386015:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626884.386017:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626884.386019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626884.386022:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11218 00000100:00080000:1.0:1778626884.386029:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626884.386034:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626884.386038:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626884.386046:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828446208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626884.386053:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626884.386056:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626884.386057:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626884.386060:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11218 00000100:00080000:1.0:1778626884.386061:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626884.386066:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626884.386073:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626884.386074:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828445824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626884.386082:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828446336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626884.386184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626884.386224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626884/real 1778626884] req@ffff954350196a00 x1865013828445824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626884.386242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828445824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626884.386264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626884.386267:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626884.386271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828445824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626884.386277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626884.386295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626884.386303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626884/real 1778626884] req@ffff954350194a80 x1865013828445952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626884.386311:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828445952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626884.386317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626884.386319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626884.386321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828445952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626884.386325:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626884.386335:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626884.386341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626884/real 1778626884] req@ffff954350195c00 x1865013828446080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626884.386348:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828446080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626884.386353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626884.386355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626884.386357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626884.386390:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626884.386404:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626884.386414:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626884/real 1778626884] req@ffff954350195500 x1865013828446208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626884.386427:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828446208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626884.386436:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626884.386440:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626884.386444:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626884.386451:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626884.386471:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626884.386481:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626884/real 1778626884] req@ffff954350194380 x1865013828446336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626884.386489:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828446336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626884.386496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626884.386498:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626884.386500:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626884.577797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626884.577833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626885.345746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626885.601731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626885.601796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626886.369763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626886.625768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626886.625819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626887.393800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626887.649761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626887.649871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626888.417738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626888.673772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626888.673867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626889.441766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626889.505760:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626889.505769:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626889.505774:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626889.505784:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11224 00000100:00080000:1.0:1778626889.505796:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626889.505804:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626889.505839:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828446464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626889.505890:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626889.505895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626889.505898:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626889.505903:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11224 00000100:00080000:1.0:1778626889.505908:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626889.505915:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626889.505916:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828446464:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778626889.505925:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626889.505940:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828446592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626889.505948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626889.505950:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626889.505952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626889.505954:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11224 00000100:00080000:1.0:1778626889.505957:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626889.505959:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626889.505963:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626889.505970:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828446720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626889.505976:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626889.505979:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626889.505981:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778626889.505983:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626889.505986:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11224 00000100:00080000:1.0:1778626889.506009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626889.506013:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626889.506018:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626889.506032:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828446848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626889.506032:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626889/real 1778626889] req@ffff954350197800 x1865013828446464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626889.506054:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626889.506057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626889.506059:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626889.506062:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11224 00000100:00080000:1.0:1778626889.506065:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778626889.506066:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828446464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626889.506070:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626889.506075:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778626889.506085:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778626889.506087:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828446976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626889.506092:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626889.506100:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828446464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626889.506123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626889.506141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626889.506149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626889/real 1778626889] req@ffff954350194700 x1865013828446592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626889.506157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828446592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626889.506164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626889.506166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626889.506168:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626889.506172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626889.506182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626889.506187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626889/real 1778626889] req@ffff954350197480 x1865013828446720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626889.506202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828446720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626889.506208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626889.506210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626889.506212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626889.506216:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626889.506228:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626889.506234:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626889/real 1778626889] req@ffff954350196d80 x1865013828446848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626889.506241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828446848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626889.506246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626889.506248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626889.506250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626889.506253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626889.506262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626889.506267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626889/real 1778626889] req@ffff954350194000 x1865013828446976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626889.506274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828446976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626889.506288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626889.506291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626889.506293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828446976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626889.697733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626889.697855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626890.465736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626890.721752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626890.721811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626891.489824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626891.745756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626891.745824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626892.513739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626892.769714:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626892.769824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626893.537771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626893.793766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626893.793832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626894.561772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626894.625776:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626894.625784:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626894.625787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626894.625795:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11229 00000100:00080000:1.0:1778626894.625805:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626894.625811:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626894.625855:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828447104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626894.625878:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626894.625882:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626894.625883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626894.625886:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11229 00000100:00080000:1.0:1778626894.625923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626894.625927:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626894.625935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626894.625948:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828447232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626894.625958:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626894.625965:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626894.625968:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626894.625969:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828447104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626894.625973:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11229 00000100:00080000:1.0:1778626894.625982:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626894.625986:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626894.625992:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626894.626003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828447360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626894.626009:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626894.626012:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626894.626013:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626894.626022:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11229 00000100:00080000:1.0:1778626894.626025:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626894.626028:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626894.626033:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626894.626040:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828447488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626894.626047:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626894.626050:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626894.626051:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626894.626054:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11229 00000100:00080000:1.0:1778626894.626072:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626894.626076:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626894.626081:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626894.626089:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626894.626096:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828447616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626894.626145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626894/real 1778626894] req@ffff954350197100 x1865013828447104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626894.626170:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828447104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626894.626197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626894.626203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626894.626208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828447104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626894.626238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626894.626267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626894.626282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626894/real 1778626894] req@ffff954350194000 x1865013828447232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626894.626296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828447232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626894.626306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626894.626311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626894.626314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626894.626321:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626894.626343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626894.626355:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626894/real 1778626894] req@ffff954350196d80 x1865013828447360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626894.626369:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828447360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626894.626378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626894.626382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626894.626385:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626894.626400:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626894.626422:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626894.626434:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626894/real 1778626894] req@ffff954350197480 x1865013828447488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626894.626448:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828447488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626894.626457:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626894.626462:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626894.626465:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626894.626472:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626894.626493:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626894.626503:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626894/real 1778626894] req@ffff954350194700 x1865013828447616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626894.626516:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828447616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626894.626525:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626894.626529:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626894.626532:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626894.817790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626894.817895:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626895.585766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626895.841774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626895.841928:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626896.609719:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626896.865777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626896.865953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626897.633750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626897.889778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626897.889840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626898.657727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626898.913761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626898.913923:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626899.681772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626899.745745:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626899.745755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626899.745760:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626899.745769:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11234 00000100:00080000:1.0:1778626899.745780:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626899.745893:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626899.745930:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828447744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626899.745986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626899.745990:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626899.745992:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626899.745997:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11234 00000100:00080000:1.0:1778626899.746002:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626899.746007:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626899.746016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626899.746028:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828447744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626899.746032:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828447872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626899.746044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626899.746048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626899.746050:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626899.746053:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11234 00000100:00080000:1.0:1778626899.746057:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626899.746061:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626899.746066:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626899.746076:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828448000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626899.746085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626899.746089:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626899.746093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626899.746095:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626899.746101:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11234 00000100:00080000:1.0:1778626899.746112:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626899.746118:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626899.746124:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626899.746138:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626899/real 1778626899] req@ffff954350197800 x1865013828447744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626899.746139:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828448128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626899.746149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626899.746157:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626899.746161:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626899.746169:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11234 00000100:00080000:1.0:1778626899.746173:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626899.746176:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626899.746180:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626899.746184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828447744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626899.746193:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828448256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626899.746200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626899.746209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626899.746215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828447744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626899.746239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626899.746258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626899.746268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626899/real 1778626899] req@ffff954350194380 x1865013828447872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626899.746277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828447872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626899.746283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626899.746285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626899.746288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828447872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626899.746293:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626899.746311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626899.746321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626899/real 1778626899] req@ffff954350195500 x1865013828448000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626899.746343:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828448000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626899.746366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626899.746370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626899.746373:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626899.746380:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626899.746403:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626899.746412:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626899/real 1778626899] req@ffff954350195c00 x1865013828448128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626899.746420:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828448128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626899.746425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626899.746427:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626899.746430:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626899.746433:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626899.746444:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626899.746450:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626899/real 1778626899] req@ffff954350194a80 x1865013828448256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626899.746457:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828448256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626899.746462:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626899.746463:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626899.746471:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626899.937771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626899.937875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626900.705763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626900.961758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626900.961873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626901.729777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626901.985815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626901.985901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626902.753745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626903.009749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626903.009851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626903.777764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626904.033770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626904.033911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626904.801755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626904.865793:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626904.865801:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626904.865805:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626904.865813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11239 00000100:00080000:1.0:1778626904.865823:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626904.865828:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626904.865861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828448384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626904.865884:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626904.865900:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626904.865903:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626904.865906:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11239 00000100:00080000:1.0:1778626904.865910:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626904.865913:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626904.865920:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626904.865930:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828448512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626904.865942:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828448384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626904.865945:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626904.865960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626904.865962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626904.865966:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11239 00000100:00080000:1.0:1778626904.865968:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626904.865970:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626904.865974:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626904.865984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828448640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626904.865991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626904.865995:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626904.865996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626904.865999:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11239 00000100:00080000:1.0:1778626904.866002:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626904.866005:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626904.866013:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626904.866021:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828448768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626904.866027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626904.866029:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626904.866031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626904.866035:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11239 00000100:00080000:1.0:1778626904.866039:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626904.866043:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626904.866047:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626904.866059:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828448896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626904.866083:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626904.866110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626904.866125:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626904.866146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626905.057790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626905.057846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626905.825756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626906.081780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626906.081880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626906.849787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626907.105721:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626907.105841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626907.873813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778626907.938080:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778626907.938141:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778626907.938165:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626907.938212:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626907.938223:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626907.938231:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778626907.938237:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778626907.938297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626904/real 1778626907] req@ffff954350194e00 x1865013828448512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626907.938329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828448512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626907.938344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626907.938350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626907.938355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626907.938364:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626904/real 1778626907] req@ffff954350195880 x1865013828448640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626907.938376:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828448640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626907.938383:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626907.938387:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626907.938390:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626907.938429:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626904/real 1778626907] req@ffff954350197b80 x1865013828448768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626907.938444:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828448768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626907.938453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626907.938457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626907.938460:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626907.938469:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626904/real 1778626907] req@ffff954350195180 x1865013828448896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626907.938480:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828448896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626907.938489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626907.938492:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626907.938495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828448896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626907.938502:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626904/real 1778626907] req@ffff954350196a00 x1865013828448384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626907.938515:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828448384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626907.938522:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626907.938525:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626907.938528:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828448384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626908.129808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626908.129883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626908.897792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626909.153824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626909.153891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626909.921802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626909.985829:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626909.985841:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626909.985846:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626909.985856:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11244 00000100:00080000:2.0:1778626909.985870:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626909.985877:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626909.985924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828449024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626909.985959:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626909.985983:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626909.985986:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626909.985992:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11244 00000100:00080000:2.0:1778626909.985998:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626909.986004:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626909.986016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828449024:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778626909.986018:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626909.986055:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828449152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626909.986069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626909.986073:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626909.986075:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626909.986079:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11244 00000100:00080000:2.0:1778626909.986085:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626909.986089:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626909.986095:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626909.986108:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828449280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626909.986125:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626909.986126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626909.986136:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626909.986139:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626909.986145:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11244 00000100:00080000:2.0:1778626909.986150:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626909.986155:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626909.986168:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626909.986181:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626909/real 1778626909] req@ffff954358789180 x1865013828449024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626909.986184:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828449408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626909.986206:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626909.986210:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626909.986212:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626909.986216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828449024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626909.986220:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11244 00000100:00080000:2.0:1778626909.986227:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626909.986235:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626909.986238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1778626909.986240:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778626909.986247:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626909.986255:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828449536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626909.986261:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828449024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626909.986309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626909.986339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626909.986354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626909/real 1778626909] req@ffff954358789c00 x1865013828449152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626909.986368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828449152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626909.986378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626909.986383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626909.986386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626909.986393:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626909.986415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626909.986427:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626909/real 1778626909] req@ffff95435878b800 x1865013828449280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626909.986449:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828449280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626909.986464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626909.986468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626909.986470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626909.986477:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626909.986500:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626909.986512:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626909/real 1778626909] req@ffff95435878a300 x1865013828449408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626909.986525:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828449408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626909.986534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626909.986551:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626909.986556:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626909.986562:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626909.986595:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626909.986608:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626909/real 1778626909] req@ffff95435878ad80 x1865013828449536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626909.986622:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828449536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626909.986646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626909.986650:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626909.986653:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626910.177811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626910.177893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626910.945844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626911.201842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626911.201981:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626911.969855:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626912.225815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626912.225887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626912.993802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626913.249799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626913.249872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626914.017793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626914.273781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626914.273862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626915.041804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626915.105768:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626915.105778:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626915.105783:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626915.105793:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11249 00000100:00080000:2.0:1778626915.105830:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626915.105837:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626915.105877:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828449664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626915.105917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626915.105921:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626915.105923:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626915.105928:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11249 00000100:00080000:2.0:1778626915.105932:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626915.105937:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626915.105953:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626915.105964:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828449792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626915.105972:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626915.105975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626915.105976:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626915.105979:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11249 00000100:00080000:2.0:1778626915.105982:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626915.105985:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626915.105988:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626915.105997:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828449920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626915.106004:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626915.106012:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626915.106016:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778626915.106017:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828449664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626915.106027:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11249 00000100:00080000:2.0:1778626915.106031:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626915.106034:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626915.106039:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626915.106050:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828450048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626915.106057:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626915.106065:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626915.106066:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626915.106069:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11249 00000100:00080000:2.0:1778626915.106072:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626915.106075:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626915.106079:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626915.106087:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828450176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626915.106093:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626915.106136:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626915/real 1778626915] req@ffff954358789f80 x1865013828449664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626915.106151:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828449664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626915.106160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626915.106164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626915.106167:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828449664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626915.106173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626915.106190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626915.106197:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626915/real 1778626915] req@ffff95435878a680 x1865013828449792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626915.106205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828449792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626915.106212:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626915.106222:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626915.106225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626915.106256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626915.106271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626915.106279:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626915/real 1778626915] req@ffff95435878ad80 x1865013828449920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626915.106294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828449920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626915.106301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626915.106303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626915.106305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828449920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626915.106309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626915.106321:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626915.106327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626915/real 1778626915] req@ffff95435878a300 x1865013828450048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626915.106337:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828450048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626915.106346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626915.106350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626915.106353:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626915.106359:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626915.106380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626915.106392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626915/real 1778626915] req@ffff95435878b800 x1865013828450176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626915.106410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828450176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626915.106417:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626915.106419:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626915.106422:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626915.297805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626915.297888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626916.065769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626916.321808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626916.321840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626917.089792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778626917.345816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626917.345832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778626918.113775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626918.369773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626918.369843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626919.137767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626919.393762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626919.393834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626920.161744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626920.225802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626920.225811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626920.225816:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626920.225826:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11254 00000100:00080000:1.0:1778626920.225839:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626920.225846:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626920.225883:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828450304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626920.225911:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626920.225916:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626920.225918:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626920.225922:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11254 00000100:00080000:1.0:1778626920.225943:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626920.225947:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626920.225959:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626920.225973:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828450432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626920.225981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626920.225984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626920.225985:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626920.225988:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11254 00000100:00080000:1.0:1778626920.225991:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626920.225994:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626920.225999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626920.226006:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828450560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626920.226013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626920.226015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626920.226016:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626920.226019:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11254 00000100:00080000:1.0:1778626920.226021:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626920.226025:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626920.226029:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626920.226046:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828450688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626920.226049:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828450304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626920.226087:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626920.226091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626920.226093:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626920.226097:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11254 00000100:00080000:1.0:1778626920.226100:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626920.226103:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626920.226109:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626920.226122:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828450816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626920.226155:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626920.226236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626920/real 1778626920] req@ffff954350195f80 x1865013828450304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626920.226258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828450304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626920.226270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626920.226275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626920.226280:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828450304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626920.226288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626920.226311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626920.226322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626920/real 1778626920] req@ffff954350196a00 x1865013828450432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626920.226352:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828450432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626920.226361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626920.226364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626920.226368:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626920.226373:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626920.226424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626920.226454:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626920/real 1778626920] req@ffff954350195180 x1865013828450560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626920.226465:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828450560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626920.226472:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626920.226476:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626920.226478:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626920.226508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626920.226525:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626920.226533:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626920/real 1778626920] req@ffff954350197b80 x1865013828450688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626920.226543:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828450688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626920.226563:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626920.226566:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626920.226569:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626920.226574:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626920.226592:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626920.226600:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626920/real 1778626920] req@ffff954350195880 x1865013828450816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626920.226610:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828450816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626920.226618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626920.226620:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626920.226623:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828450816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626920.417737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626920.417828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626921.185761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626921.441767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626921.441786:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626922.209732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626922.465767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626922.465781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626923.233763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626923.489807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626923.489850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626924.257774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626924.513755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626924.513815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626925.281754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626925.345840:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626925.345851:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626925.345857:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626925.345868:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11259 00000100:00080000:1.0:1778626925.345881:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626925.345889:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626925.345936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828450944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626925.345960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626925.345963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626925.345965:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626925.345968:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11259 00000100:00080000:1.0:1778626925.345970:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626925.345973:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626925.346000:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626925.346010:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828451072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626925.346020:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828450944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626925.346021:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626925.346027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626925.346030:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626925.346034:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11259 00000100:00080000:1.0:1778626925.346037:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626925.346039:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626925.346042:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626925.346051:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828451200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626925.346058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626925.346060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626925.346061:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626925.346064:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11259 00000100:00080000:1.0:1778626925.346066:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626925.346068:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626925.346071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626925.346079:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828451328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626925.346085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626925.346088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626925.346095:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626925.346098:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11259 00000100:00080000:1.0:1778626925.346104:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626925.346107:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626925.346111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778626925.346122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626925.346122:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828451456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626925.346171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626925/real 1778626925] req@ffff954350194e00 x1865013828450944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626925.346192:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828450944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626925.346217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626925.346221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626925.346226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828450944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626925.346256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626925.346295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626925.346310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626925/real 1778626925] req@ffff954350194a80 x1865013828451072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626925.346324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828451072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626925.346334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626925.346338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626925.346341:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626925.346347:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626925.346368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626925.346379:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626925/real 1778626925] req@ffff954350195c00 x1865013828451200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626925.346392:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828451200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626925.346401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626925.346404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626925.346407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626925.346422:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626925.346447:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626925.346459:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626925/real 1778626925] req@ffff954350195500 x1865013828451328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626925.346472:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828451328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626925.346493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626925.346497:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626925.346501:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626925.346508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626925.346529:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626925.346539:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626925/real 1778626925] req@ffff954350194380 x1865013828451456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626925.346551:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828451456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626925.346561:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626925.346565:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626925.346568:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626925.537805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626925.537842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626926.305766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626926.561744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626926.561762:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626927.329732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626927.585754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626927.585778:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626928.353790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626928.609774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626928.609800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626929.377758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626929.633781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626929.633802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626930.401797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626930.465815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626930.465826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626930.465831:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626930.465841:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11264 00000100:00080000:2.0:1778626930.465854:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626930.465860:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626930.465904:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828451584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626930.465955:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626930.465958:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626930.465960:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626930.465969:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11264 00000100:00080000:2.0:1778626930.465972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626930.465976:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626930.465985:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626930.465995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828451712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626930.466003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626930.466006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626930.466007:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626930.466010:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11264 00000100:00080000:2.0:1778626930.466012:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626930.466014:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626930.466017:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626930.466023:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828451584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626930.466027:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828451840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626930.466037:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626930.466039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626930.466040:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626930.466043:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11264 00000100:00080000:2.0:1778626930.466051:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626930.466054:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626930.466059:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626930.466069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828451968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626930.466076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626930.466079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626930.466080:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626930.466082:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11264 00000100:00080000:2.0:1778626930.466085:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626930.466087:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626930.466089:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626930.466096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828452096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626930.466122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626930.466167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626930/real 1778626930] req@ffff954358789c00 x1865013828451584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626930.466203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828451584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626930.466215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626930.466220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626930.466225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828451584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626930.466233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626930.466259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626930.466272:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626930/real 1778626930] req@ffff954358789180 x1865013828451712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626930.466284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828451712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626930.466293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626930.466297:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626930.466300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626930.466330:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626930.466352:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626930.466363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626930/real 1778626930] req@ffff954358788e00 x1865013828451840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626930.466393:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828451840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626930.466403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626930.466406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626930.466410:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626930.466417:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626930.466440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626930.466452:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626930/real 1778626930] req@ffff954358789500 x1865013828451968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626930.466465:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828451968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626930.466473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626930.466477:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626930.466480:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828451968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626930.466487:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626930.466506:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626930.466517:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626930/real 1778626930] req@ffff954358788a80 x1865013828452096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626930.466529:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828452096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626930.466562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626930.466566:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626930.466570:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626930.657799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626930.657814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626931.425746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626931.681755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626931.681776:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626932.449812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626932.705810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626932.705857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626933.473774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626933.729777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626933.729810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626934.497852:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778626934.753757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626934.753795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626935.521759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626935.585842:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626935.585850:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626935.585854:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626935.585862:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11270 00000100:00080000:1.0:1778626935.585872:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626935.585877:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626935.585910:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828452224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626935.585939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626935.585941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626935.585943:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626935.585946:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11270 00000100:00080000:1.0:1778626935.585948:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626935.585950:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626935.585957:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626935.585967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828452352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626935.585995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626935.586000:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778626935.586002:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828452224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626935.586002:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626935.586014:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11270 00000100:00080000:1.0:1778626935.586019:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626935.586022:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626935.586026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626935.586037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828452480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626935.586043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626935.586046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626935.586048:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626935.586051:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11270 00000100:00080000:1.0:1778626935.586054:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626935.586056:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626935.586059:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626935.586065:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828452608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626935.586071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626935.586073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626935.586075:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626935.586090:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11270 00000100:00080000:1.0:1778626935.586099:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778626935.586105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626935.586106:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626935.586113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626935.586126:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828452736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626935.586156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626935/real 1778626935] req@ffff954350197800 x1865013828452224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626935.586181:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828452224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626935.586190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626935.586194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626935.586198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828452224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626935.586221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626935.586240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626935.586249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626935/real 1778626935] req@ffff954350194700 x1865013828452352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626935.586257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828452352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626935.586275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626935.586278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626935.586280:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626935.586285:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626935.586298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626935.586305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626935/real 1778626935] req@ffff954350197480 x1865013828452480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626935.586313:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828452480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626935.586318:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626935.586320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626935.586321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626935.586326:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626935.586337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626935.586342:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626935/real 1778626935] req@ffff954350196d80 x1865013828452608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626935.586350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828452608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626935.586355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626935.586357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626935.586363:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626935.586368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626935.586381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626935.586388:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626935/real 1778626935] req@ffff954350194000 x1865013828452736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626935.586395:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828452736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626935.586401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626935.586403:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626935.586405:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626935.777819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626935.777865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626936.545785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626936.802167:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626936.802371:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626937.569742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626937.825849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626937.825873:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778626938.593789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778626938.849925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626938.849990:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778626939.617771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626939.873749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626939.873846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626940.641736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626940.705832:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626940.705844:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626940.705849:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626940.705860:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11275 00000100:00080000:2.0:1778626940.705872:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626940.705878:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626940.705914:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828452864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626940.705942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626940.705945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626940.705947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626940.705951:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11275 00000100:00080000:2.0:1778626940.705954:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626940.705956:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626940.705964:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626940.705975:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828452992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626940.706045:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828452864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626940.706053:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626940.706057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626940.706058:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626940.706062:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11275 00000100:00080000:2.0:1778626940.706080:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626940.706101:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626940.706116:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778626940.706128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626940.706134:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828453120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626940.706145:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626940.706148:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626940.706150:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626940.706156:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11275 00000100:00080000:2.0:1778626940.706162:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626940.706165:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626940.706170:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626940.706171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626940/real 1778626940] req@ffff954358789880 x1865013828452864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626940.706184:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828453248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626940.706194:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626940.706195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828452864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626940.706201:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626940.706205:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626940.706215:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11275 00000100:00080000:2.0:1778626940.706220:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626940.706221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626940.706225:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626940.706231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:2.0:1778626940.706233:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626940.706238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828452864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626940.706246:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828453376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626940.706266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626940.706305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626940.706319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626940/real 1778626940] req@ffff954358788700 x1865013828452992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626940.706337:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828452992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626940.706345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626940.706348:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626940.706351:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828452992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626940.706356:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626940.706371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626940.706379:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626940/real 1778626940] req@ffff95435878b100 x1865013828453120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626940.706388:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828453120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626940.706414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626940.706417:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626940.706419:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626940.706423:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626940.706436:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626940.706443:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626940/real 1778626940] req@ffff95435878bb80 x1865013828453248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626940.706462:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828453248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626940.706469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626940.706471:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626940.706473:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626940.706487:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626940.706510:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626940.706518:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626940/real 1778626940] req@ffff95435878b480 x1865013828453376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626940.706527:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828453376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626940.706534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626940.706536:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626940.706538:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626940.897753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626940.897912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626941.665742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626941.921774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626941.922037:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626942.689797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626942.945774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626942.945834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626943.713774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626943.969755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626943.969835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626944.737818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626944.993812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626944.993906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626945.761746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626945.825767:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626945.825776:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626945.825779:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626945.825787:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11280 00000100:00080000:2.0:1778626945.825797:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626945.825803:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626945.825833:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828453504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626945.825857:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626945.825861:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626945.825863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626945.825867:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11280 00000100:00080000:2.0:1778626945.825871:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626945.825874:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626945.825896:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626945.825911:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828453632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626945.825918:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828453504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626945.825929:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626945.825933:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626945.825949:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626945.825953:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11280 00000100:00080000:2.0:1778626945.825957:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626945.825960:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626945.825965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778626945.825975:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626945.825978:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828453760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626945.825987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626945.825989:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626945.825991:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626945.825994:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11280 00000100:00080000:2.0:1778626945.825997:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626945.825999:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626945.826003:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626945.826010:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828453888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626945.826015:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626945/real 1778626945] req@ffff954358788380 x1865013828453504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626945.826020:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626945.826028:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626945.826031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626945.826038:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11280 00000100:00080000:2.0:1778626945.826042:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626945.826044:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626945.826047:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626945.826054:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828453504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626945.826061:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828454016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626945.826099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626945.826106:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626945.826110:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828453504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626945.826136:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626945.826160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626945.826171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626945/real 1778626945] req@ffff95435878aa00 x1865013828453632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626945.826182:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828453632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626945.826191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626945.826194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626945.826198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626945.826204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626945.826225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626945.826236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626945/real 1778626945] req@ffff95435878b480 x1865013828453760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626945.826247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828453760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626945.826253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626945.826255:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626945.826257:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626945.826267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626945.826283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626945.826290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626945/real 1778626945] req@ffff95435878bb80 x1865013828453888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626945.826297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828453888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626945.826303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626945.826305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626945.826307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828453888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626945.826310:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626945.826319:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626945.826324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626945/real 1778626945] req@ffff95435878b100 x1865013828454016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626945.826330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828454016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626945.826336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626945.826338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626945.826340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626946.017735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626946.017807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626946.785732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626947.041769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626947.041830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626947.809767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626948.065743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626948.065879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626948.833762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626949.089741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626949.089843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626949.857765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626950.113774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626950.113867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626950.881771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626950.945833:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626950.945841:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626950.945844:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626950.945852:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11285 00000100:00080000:2.0:1778626950.945862:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626950.945868:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626950.945904:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828454144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626950.945935:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626950.945939:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626950.945941:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626950.945944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11285 00000100:00080000:2.0:1778626950.945948:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626950.945950:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626950.945958:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626950.945969:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828454272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626950.945978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626950.945986:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828454144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626950.945995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626950.945997:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626950.946001:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11285 00000100:00080000:2.0:1778626950.946004:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626950.946007:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626950.946010:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626950.946021:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828454400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626950.946028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626950.946030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626950.946031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626950.946033:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11285 00000100:00080000:2.0:1778626950.946036:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626950.946038:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626950.946041:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626950.946048:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626950.946052:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828454528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626950.946059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626950.946062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626950.946064:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626950.946067:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11285 00000100:00080000:2.0:1778626950.946069:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626950.946071:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626950.946079:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626950.946082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626950/real 1778626950] req@ffff954358788700 x1865013828454144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626950.946089:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828454656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626950.946113:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828454144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626950.946122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626950.946125:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626950.946129:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828454144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626950.946153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626950.946176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626950.946189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626950/real 1778626950] req@ffff954358789880 x1865013828454272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626950.946203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828454272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626950.946210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626950.946213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626950.946215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626950.946219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626950.946234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626950.946241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626950/real 1778626950] req@ffff954358788a80 x1865013828454400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626950.946249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828454400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626950.946262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626950.946265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626950.946267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626950.946271:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626950.946284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626950.946290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626950/real 1778626950] req@ffff954358789500 x1865013828454528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626950.946297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828454528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626950.946302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626950.946304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626950.946306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626950.946309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626950.946318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626950.946322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626950/real 1778626950] req@ffff954358788e00 x1865013828454656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626950.946329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828454656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626950.946334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626950.946336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626950.946342:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626951.137748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626951.137826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626951.905810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626952.161853:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626952.161919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626952.929800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626953.185765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626953.185869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626953.953800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626954.209838:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626954.209940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626954.977790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626955.233797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626955.233914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626956.001769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626956.065785:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626956.065793:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626956.065798:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626956.065805:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11290 00000100:00080000:1.0:1778626956.065815:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626956.065837:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626956.065865:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828454784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626956.065886:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626956.065889:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626956.065890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626956.065893:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11290 00000100:00080000:1.0:1778626956.065896:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626956.065898:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626956.065905:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626956.065915:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828454912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626956.065923:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626956.065928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778626956.065930:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828454784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626956.065930:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626956.065938:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11290 00000100:00080000:1.0:1778626956.065940:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626956.065942:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626956.065945:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626956.065956:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828455040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626956.065963:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626956.065965:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626956.065971:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626956.065974:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11290 00000100:00080000:1.0:1778626956.065985:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626956.065988:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626956.065992:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626956.066003:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778626956.066003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828455168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626956.066013:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626956.066015:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626956.066017:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626956.066020:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11290 00000100:00080000:1.0:1778626956.066022:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626956.066025:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626956.066028:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626956.066034:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828455296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626956.066051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626956/real 1778626956] req@ffff954350197100 x1865013828454784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626956.066071:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828454784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626956.066082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626956.066086:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626956.066090:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828454784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626956.066112:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626956.066135:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626956.066147:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626956/real 1778626956] req@ffff954350194000 x1865013828454912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626956.066159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828454912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626956.066167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626956.066170:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626956.066173:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828454912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626956.066179:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626956.066195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626956.066223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626956/real 1778626956] req@ffff954350196d80 x1865013828455040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626956.066235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828455040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626956.066242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626956.066245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626956.066249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626956.066254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626956.066272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626956.066281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626956/real 1778626956] req@ffff954350197480 x1865013828455168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626956.066292:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828455168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626956.066300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626956.066303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626956.066306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626956.066311:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626956.066327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626956.066336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626956/real 1778626956] req@ffff954350194700 x1865013828455296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626956.066345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828455296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626956.066359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626956.066362:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626956.066365:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626956.257793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626956.257850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626957.025751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626957.281779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626957.281846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626958.049770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626958.305819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626958.305903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626959.073786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626959.329790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626959.329871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626960.097777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626960.353806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626960.353893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626961.121777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626961.185819:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626961.185834:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626961.185840:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626961.185853:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11295 00000100:00080000:1.0:1778626961.185892:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626961.185909:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626961.185953:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828455424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626961.185986:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626961.185997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626961.186000:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626961.186005:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11295 00000100:00080000:1.0:1778626961.186009:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626961.186014:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626961.186027:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626961.186038:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828455424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626961.186044:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828455552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626961.186058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626961.186062:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626961.186064:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626961.186068:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11295 00000100:00080000:1.0:1778626961.186073:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626961.186077:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626961.186083:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626961.186095:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828455680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626961.186106:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778626961.186114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626961.186119:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626961.186122:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626961.186128:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11295 00000100:00080000:1.0:1778626961.186134:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626961.186139:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626961.186146:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626961.186161:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626961/real 1778626961] req@ffff954350197800 x1865013828455424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778626961.186162:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828455808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626961.186180:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626961.186185:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626961.186187:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626961.186192:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11295 00000100:00100000:0.0:1778626961.186196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828455424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778626961.186199:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626961.186208:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778626961.186212:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778626961.186214:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778626961.186218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626961.186223:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828455424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778626961.186232:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828455936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626961.186249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626961.186268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626961.186277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626961/real 1778626961] req@ffff954350194380 x1865013828455552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626961.186286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828455552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626961.186293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626961.186295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626961.186297:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626961.186309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626961.186323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626961.186330:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626961/real 1778626961] req@ffff954350195500 x1865013828455680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626961.186339:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828455680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626961.186344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626961.186346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626961.186349:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626961.186352:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626961.186364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626961.186369:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626961/real 1778626961] req@ffff954350195c00 x1865013828455808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626961.186376:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828455808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626961.186382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626961.186384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626961.186385:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626961.186389:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626961.186398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626961.186403:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626961/real 1778626961] req@ffff954350194a80 x1865013828455936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626961.186416:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828455936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626961.186422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626961.186424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626961.186427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828455936:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778626961.377803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778626961.377837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778626962.145810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626962.401751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626962.401845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626963.169792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626963.425776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626963.425886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626964.193850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626964.449739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626964.449806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626965.217819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626965.473726:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626965.473829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626966.241798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626966.305759:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626966.305769:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626966.305795:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626966.305806:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11300 00000100:00080000:1.0:1778626966.305817:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626966.305833:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626966.305862:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828456064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626966.305890:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626966.305893:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626966.305894:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626966.305897:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11300 00000100:00080000:1.0:1778626966.305900:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626966.305903:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626966.305910:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626966.305919:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828456192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626966.305926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626966.305928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626966.305930:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626966.305932:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11300 00000100:00080000:1.0:1778626966.305934:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626966.305939:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626966.305941:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828456064:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778626966.305942:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626966.305953:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828456320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626966.305971:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626966.305973:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626966.305975:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626966.305978:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11300 00000100:00080000:1.0:1778626966.305982:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626966.305984:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626966.305988:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626966.305997:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828456448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626966.306005:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778626966.306010:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626966.306013:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626966.306014:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626966.306017:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11300 00000100:00080000:1.0:1778626966.306019:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626966.306022:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626966.306025:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626966.306032:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828456576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626966.306053:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626966/real 1778626966] req@ffff954350194e00 x1865013828456064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626966.306072:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828456064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626966.306082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626966.306087:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626966.306091:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828456064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626966.306108:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626966.306135:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626966.306148:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626966/real 1778626966] req@ffff954350195880 x1865013828456192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626966.306162:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828456192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626966.306171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626966.306175:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626966.306178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626966.306205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626966.306227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626966.306239:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626966/real 1778626966] req@ffff954350197b80 x1865013828456320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626966.306252:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828456320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626966.306261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626966.306264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626966.306267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626966.306274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626966.306296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626966.306307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626966/real 1778626966] req@ffff954350195180 x1865013828456448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626966.306329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828456448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626966.306339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626966.306342:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626966.306346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626966.306353:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626966.306373:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626966.306384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626966/real 1778626966] req@ffff954350196a00 x1865013828456576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626966.306397:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828456576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626966.306407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626966.306410:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626966.306414:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626966.497850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626966.497886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626967.265797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626967.521831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626967.521936:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626968.289755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626968.545786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626968.545901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626969.313785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626969.569823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626969.569916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626970.337809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626970.593822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626970.593957:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626971.361795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778626971.425788:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626971.425801:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778626971.425806:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626971.425819:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11305 00000100:00080000:0.0:1778626971.425835:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626971.425843:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626971.425888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828456704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626971.425915:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626971.425918:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778626971.425919:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626971.425923:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11305 00000100:00080000:0.0:1778626971.425926:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626971.425928:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626971.425938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626971.425949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828456832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626971.426004:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778626971.426010:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828456704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778626971.426011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778626971.426019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626971.426026:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11305 00000100:00080000:0.0:1778626971.426030:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626971.426034:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626971.426040:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778626971.426053:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828456960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626971.426063:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626971.426065:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778626971.426067:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626971.426070:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11305 00000100:00080000:0.0:1778626971.426077:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626971.426080:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626971.426084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778626971.426092:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828457088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626971.426099:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778626971.426102:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778626971.426104:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778626971.426109:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11305 00000100:00080000:0.0:1778626971.426118:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778626971.426121:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778626971.426126:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626971.426129:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626971.426137:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828457216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626971.426189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626971.426207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626971.426226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626971.617806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626971.617987:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626972.385759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626972.641794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626972.641930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626973.409839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626973.665726:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626973.666051:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626974.433743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778626974.497974:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778626974.498034:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778626974.498051:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626974.498104:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626974.498115:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626974.498123:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778626974.498130:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778626974.498238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626971/real 1778626974] req@ffff954342f49c00 x1865013828457216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626974.498265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828457216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626974.498276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626974.498283:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626974.498287:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626974.498294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626971/real 1778626974] req@ffff954342f49880 x1865013828456832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626974.498305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828456832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626974.498330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626974.498334:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626974.498337:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626974.498343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626971/real 1778626974] req@ffff954342f4a300 x1865013828456960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626974.498372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828456960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626974.498380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626974.498384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626974.498386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828456960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626974.498392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626971/real 1778626974] req@ffff954342f4b480 x1865013828457088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626974.498403:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828457088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626974.498410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626974.498413:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626974.498415:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626974.498422:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626971/real 1778626974] req@ffff954342f4bb80 x1865013828456704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626974.498434:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828456704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778626996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626974.498441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626974.498445:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626974.498448:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828456704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778626974.689747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626974.689904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626975.457758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626975.713770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626975.713903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626976.481732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778626976.545868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626976.545880:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778626976.545885:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626976.545896:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11311 00000100:00080000:1.0:1778626976.545910:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626976.545916:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778626976.545955:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828457344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626976.545984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626976.545988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778626976.545989:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626976.545993:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11311 00000100:00080000:1.0:1778626976.545995:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626976.545998:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626976.546007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626976.546045:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828457472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626976.546053:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626976.546056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778626976.546057:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626976.546060:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11311 00000100:00080000:1.0:1778626976.546064:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626976.546067:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626976.546071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778626976.546079:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828457600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778626976.546088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828457344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778626976.546089:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626976.546094:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778626976.546097:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626976.546103:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11311 00000100:00080000:1.0:1778626976.546105:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626976.546108:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626976.546111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778626976.546118:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828457728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778626976.546126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778626976.546130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778626976.546132:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778626976.546143:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11311 00000100:00080000:1.0:1778626976.546147:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778626976.546153:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778626976.546162:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778626976.546174:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828457856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778626976.546196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626976.546246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626976/real 1778626976] req@ffff954350195f80 x1865013828457344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626976.546291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828457344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626976.546305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626976.546309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626976.546313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828457344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778626976.546319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626976.546345:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626976.546361:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626976/real 1778626976] req@ffff954350196a00 x1865013828457472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626976.546374:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828457472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626976.546383:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626976.546387:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626976.546390:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626976.546424:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778626976.546446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626976.546468:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626976/real 1778626976] req@ffff954350195180 x1865013828457600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626976.546481:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828457600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626976.546489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626976.546492:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626976.546495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778626976.546501:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626976.546523:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626976.546534:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626976/real 1778626976] req@ffff954350197b80 x1865013828457728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626976.546546:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828457728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626976.546553:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626976.546557:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626976.546561:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778626976.546566:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778626976.546584:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626976.546594:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626976/real 1778626976] req@ffff954350195880 x1865013828457856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778626976.546606:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828457856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626976.546624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778626976.546628:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778626976.546632:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828457856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626976.737741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626976.737901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626977.505797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626977.761773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626977.761896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626978.529731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626978.785738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626978.785842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626979.553764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626979.809756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626979.809814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626980.577748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626980.833736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626980.833796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626981.601773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626981.665787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626981.665796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626981.665799:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626981.665808:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11316 00000100:00080000:2.0:1778626981.665818:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626981.665824:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626981.665875:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828457984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626981.665917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626981.665920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626981.665921:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626981.665925:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11316 00000100:00080000:2.0:1778626981.665927:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626981.665945:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626981.665954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626981.665982:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828458112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626981.665986:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828457984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626981.666000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626981.666003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778626981.666005:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626981.666008:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11316 00000100:00080000:2.0:1778626981.666012:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626981.666015:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626981.666019:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626981.666045:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828458240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626981.666053:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626981.666055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626981.666056:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626981.666059:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11316 00000100:00080000:2.0:1778626981.666062:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626981.666065:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626981.666068:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626981.666073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778626981.666081:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828458368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626981.666090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626981.666097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626981.666098:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626981.666103:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11316 00000100:00080000:2.0:1778626981.666107:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778626981.666111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626981/real 1778626981] req@ffff954358789180 x1865013828457984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778626981.666114:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626981.666130:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626981.666131:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828457984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626981.666143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778626981.666145:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828458496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626981.666149:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626981.666157:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828457984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626981.666201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626981.666220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626981.666229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626981/real 1778626981] req@ffff954358789c00 x1865013828458112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626981.666238:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828458112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626981.666251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626981.666254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626981.666264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626981.666269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626981.666284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626981.666292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626981/real 1778626981] req@ffff95435878b800 x1865013828458240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626981.666302:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828458240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626981.666311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626981.666314:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626981.666318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626981.666324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626981.666347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626981.666359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626981/real 1778626981] req@ffff95435878a300 x1865013828458368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626981.666373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828458368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626981.666382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626981.666386:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626981.666389:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626981.666394:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626981.666411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626981.666425:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626981/real 1778626981] req@ffff95435878ad80 x1865013828458496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626981.666434:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828458496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626981.666440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626981.666442:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626981.666444:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626981.857773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626981.857847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626982.625776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626982.881745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626982.881787:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626983.649750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626983.905745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626983.905808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626984.673751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626984.929746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626984.929816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626985.697761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626985.953754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626985.953818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626986.721750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626986.785747:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626986.785755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626986.785759:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626986.785767:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11321 00000100:00080000:3.0:1778626986.785794:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626986.785802:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626986.785837:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828458624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626986.785876:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626986.785879:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626986.785881:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626986.785886:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11321 00000100:00080000:3.0:1778626986.785890:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626986.785894:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626986.785903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626986.785917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828458752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626986.785928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778626986.785931:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828458624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778626986.785932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626986.785933:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626986.785937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11321 00000100:00080000:3.0:1778626986.785939:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626986.785943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626986.785948:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626986.785956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828458880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626986.785963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626986.785983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626986.785986:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626986.785990:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11321 00000100:00080000:3.0:1778626986.785995:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626986.785999:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626986.786005:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778626986.786014:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778626986.786020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828459008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626986.786031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626986.786035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626986.786038:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626986.786043:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11321 00000100:00100000:0.0:1778626986.786045:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626986/real 1778626986] req@ffff95434701d180 x1865013828458624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778626986.786049:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626986.786059:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626986.786065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778626986.786080:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828458624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626986.786090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778626986.786093:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828459136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778626986.786096:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626986.786103:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828458624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626986.786129:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626986.786147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626986.786155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626986/real 1778626986] req@ffff95434701f800 x1865013828458752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626986.786163:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828458752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626986.786179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626986.786182:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626986.786184:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626986.786188:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626986.786200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626986.786206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626986/real 1778626986] req@ffff95434701d880 x1865013828458880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626986.786213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828458880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626986.786218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626986.786220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626986.786222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828458880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626986.786225:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626986.786235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626986.786241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626986/real 1778626986] req@ffff95434701ea00 x1865013828459008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626986.786248:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828459008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626986.786252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626986.786254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626986.786260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626986.786264:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626986.786276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626986.786282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626986/real 1778626986] req@ffff95434701c000 x1865013828459136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626986.786289:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828459136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626986.786294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626986.786296:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626986.786298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626986.977771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626986.977836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626987.745737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626988.001753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626988.001816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626988.769735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626989.025770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626989.025845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626989.793822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626990.049817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626990.049889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626990.817739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626991.073746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626991.073817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626991.841794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778626991.905845:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626991.905878:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778626991.905884:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626991.905895:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11326 00000100:00080000:2.0:1778626991.905907:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626991.905913:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778626991.905984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828459264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626991.906012:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626991.906015:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778626991.906019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626991.906023:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11326 00000100:00080000:2.0:1778626991.906026:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626991.906029:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626991.906039:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626991.906050:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828459392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626991.906058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626991.906063:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778626991.906064:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828459264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778626991.906065:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626991.906070:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11326 00000100:00080000:2.0:1778626991.906074:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626991.906077:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626991.906080:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778626991.906106:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828459520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626991.906119:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626991.906121:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778626991.906123:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626991.906127:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11326 00000100:00080000:2.0:1778626991.906130:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626991.906133:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626991.906137:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778626991.906145:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828459648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778626991.906152:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778626991.906155:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778626991.906156:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778626991.906160:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11326 00000800:00000100:0.0:1778626991.906163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778626991.906164:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778626991.906168:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778626991.906171:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778626991.906178:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828459776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626991.906220:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626991/real 1778626991] req@ffff95435878a680 x1865013828459264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626991.906237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828459264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626991.906247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626991.906251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626991.906254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828459264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626991.906291:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626991.906316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626991.906339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626991/real 1778626991] req@ffff954358789f80 x1865013828459392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626991.906361:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828459392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626991.906368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626991.906371:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626991.906373:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626991.906378:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626991.906409:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626991.906430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626991/real 1778626991] req@ffff95435878ad80 x1865013828459520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626991.906444:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828459520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626991.906452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626991.906456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626991.906458:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626991.906463:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626991.906481:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626991.906488:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626991/real 1778626991] req@ffff95435878a300 x1865013828459648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626991.906498:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828459648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626991.906514:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626991.906518:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626991.906522:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626991.906528:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626991.906561:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626991.906572:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626991/real 1778626991] req@ffff95435878b800 x1865013828459776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626991.906580:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828459776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626991.906586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626991.906589:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626991.906591:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828459776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626992.097783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626992.097958:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626992.865772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626993.121825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626993.121886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626993.889744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626994.145739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778626994.145821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626994.913767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626995.169781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626995.169851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626995.937735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626996.193765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778626996.193815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626996.961784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778626997.025780:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626997.025792:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778626997.025797:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626997.025808:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11331 00000100:00080000:3.0:1778626997.025822:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626997.025829:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778626997.025891:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828459904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626997.025924:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626997.025928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778626997.025930:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626997.025934:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11331 00000100:00080000:3.0:1778626997.025937:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626997.025941:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626997.025951:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626997.025981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828460032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626997.025992:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626997.025995:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778626997.025997:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626997.026000:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11331 00000100:00080000:3.0:1778626997.026002:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626997.026005:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778626997.026025:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828459904:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778626997.026030:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778626997.026049:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828460160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778626997.026080:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626997.026085:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778626997.026087:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626997.026093:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11331 00000100:00080000:3.0:1778626997.026099:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626997.026104:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626997.026111:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778626997.026143:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828460288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778626997.026147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778626997.026157:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778626997.026160:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778626997.026162:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778626997.026166:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11331 00000100:00080000:3.0:1778626997.026170:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778626997.026173:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778626997.026176:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778626997.026187:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828460416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778626997.026202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626997/real 1778626997] req@ffff95434701d500 x1865013828459904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626997.026226:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828459904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626997.026239:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626997.026253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626997.026258:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828459904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778626997.026267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626997.026294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626997.026307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626997/real 1778626997] req@ffff95434701c380 x1865013828460032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626997.026316:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828460032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626997.026322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626997.026341:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626997.026344:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626997.026369:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778626997.026396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626997.026405:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626997/real 1778626997] req@ffff95434701f100 x1865013828460160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626997.026415:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828460160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626997.026421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626997.026439:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626997.026441:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778626997.026445:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626997.026464:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626997.026470:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626997/real 1778626997] req@ffff95434701c700 x1865013828460288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626997.026478:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828460288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626997.026484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626997.026485:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626997.026487:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778626997.026508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778626997.026521:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778626997.026528:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778626997/real 1778626997] req@ffff95434701c000 x1865013828460416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778626997.026537:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828460416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778626997.026545:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778626997.026547:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778626997.026549:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778626997.217792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778626997.217849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626997.985756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626998.241790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626998.241859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778626999.009762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778626999.265763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778626999.265891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627000.033735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627000.289770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627000.289818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627001.057820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627001.313808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627001.313880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627002.081897:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627002.145757:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627002.145782:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627002.145788:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627002.145802:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11336 00000100:00080000:3.0:1778627002.145837:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627002.145865:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627002.145939:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828460544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627002.146004:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627002.146013:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627002.146017:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627002.146027:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11336 00000100:00080000:3.0:1778627002.146037:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627002.146046:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627002.146063:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828460544:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778627002.146066:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627002.146116:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828460672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627002.146145:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627002.146148:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627002.146166:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627002.146171:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11336 00000800:00000100:0.0:1778627002.146176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627002.146196:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627002.146201:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627002.146208:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627002.146229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627002/real 1778627002] req@ffff95434701ea00 x1865013828460544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627002.146241:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828460800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627002.146250:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627002.146265:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627002.146272:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627002.146301:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11336 00000100:00080000:3.0:1778627002.146306:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627002.146311:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828460544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627002.146315:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627002.146330:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:0.0:1778627002.146335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627002.146345:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:3.0:1778627002.146346:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828460928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627002.146355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828460544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627002.146377:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627002.146380:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627002.146382:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627002.146385:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11336 00000100:00080000:3.0:1778627002.146390:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627002.146394:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627002.146396:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460672:192.168.204.102@tcp:38:kworker.0 00000008:00100000:3.0:1778627002.146402:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627002.146417:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828461056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627002.146426:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627002.146438:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627002/real 1778627002] req@ffff95434701d880 x1865013828460672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627002.146451:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828460672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627002.146458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627002.146461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627002.146464:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627002.146469:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627002.146481:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627002.146487:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627002/real 1778627002] req@ffff95434701f800 x1865013828460800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627002.146494:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828460800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627002.146509:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627002.146511:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627002.146513:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627002.146517:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627002.146531:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627002.146537:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627002/real 1778627002] req@ffff95434701d180 x1865013828460928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627002.146544:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828460928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627002.146549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627002.146551:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627002.146553:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828460928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627002.146564:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627002.146574:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627002.146579:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627002/real 1778627002] req@ffff95434701df80 x1865013828461056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627002.146586:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828461056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627002.146592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627002.146594:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627002.146599:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627002.337774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627002.337827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627003.105857:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627003.362198:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627003.362244:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778627004.129831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627004.385852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627004.385946:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627005.153768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627005.409818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627005.410490:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627006.177793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627006.433924:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627006.434166:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627007.201777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627007.265850:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627007.265862:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627007.265868:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627007.265882:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11341 00000100:00080000:1.0:1778627007.265898:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627007.265907:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627007.265960:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828461184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627007.265996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627007.266001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627007.266003:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627007.266009:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11341 00000100:00080000:1.0:1778627007.266014:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627007.266024:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627007.266048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828461184:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627007.266059:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627007.266081:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828461312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627007.266095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627007.266099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627007.266101:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627007.266105:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11341 00000100:00080000:1.0:1778627007.266112:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627007.266116:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627007.266121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627007.266123:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627007.266141:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828461440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627007.266152:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627007.266156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627007.266159:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627007.266163:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627007/real 1778627007] req@ffff954350194e00 x1865013828461184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627007.266168:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11341 00000100:00080000:1.0:1778627007.266177:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627007.266183:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627007.266188:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828461184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627007.266192:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:0.0:1778627007.266200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627007.266206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627007.266212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828461184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627007.266212:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828461568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627007.266228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627007.266232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627007.266239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461312:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627007.266244:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627007.266252:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11341 00000100:00080000:1.0:1778627007.266260:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778627007.266262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627007.266270:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627007.266275:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627007/real 1778627007] req@ffff954350194a80 x1865013828461312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627007.266277:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627007.266291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828461312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627007.266298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778627007.266300:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828461696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627007.266303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627007.266309:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627007.266319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627007.266337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627007.266354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627007/real 1778627007] req@ffff954350195c00 x1865013828461440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627007.266363:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828461440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627007.266370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627007.266372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627007.266375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627007.266379:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627007.266391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627007.266396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627007/real 1778627007] req@ffff954350195500 x1865013828461568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627007.266404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828461568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627007.266411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627007.266415:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627007.266418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627007.266434:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627007.266455:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627007.266466:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627007/real 1778627007] req@ffff954350194380 x1865013828461696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627007.266474:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828461696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627007.266486:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627007.266489:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627007.266491:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461696:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778627007.458085:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627007.458094:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778627008.225812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627008.481869:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627008.481961:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627009.249792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627009.505808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627009.505818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627010.273826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627010.529780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627010.529840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627011.297747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627011.553756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627011.553807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627012.321827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627012.385754:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627012.385762:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627012.385766:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627012.385773:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11346 00000100:00080000:0.0:1778627012.385783:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627012.385788:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627012.385818:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828461824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627012.385841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627012.385843:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627012.385845:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627012.385848:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11346 00000100:00080000:0.0:1778627012.385850:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627012.385852:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627012.385858:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627012.385866:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828461952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627012.385873:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627012.385875:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627012.385889:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627012.385892:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11346 00000100:00080000:0.0:1778627012.385895:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627012.385898:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627012.385901:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627012.385908:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828462080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627012.385915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627012.385918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627012.385919:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627012.385927:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11346 00000100:00100000:2.0:1778627012.385929:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828461824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627012.385930:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627012.385937:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627012.385949:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627012.385956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828462208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627012.385963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627012.385965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627012.385966:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627012.385970:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11346 00000100:00080000:0.0:1778627012.385973:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627012.385975:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627012.385977:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627012.385996:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828462336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627012.386025:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627012.386073:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627012/real 1778627012] req@ffff954342f49500 x1865013828461824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627012.386096:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828461824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627012.386131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627012.386137:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627012.386141:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828461824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627012.386150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627012.386179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627012.386192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627012/real 1778627012] req@ffff954342f4ad80 x1865013828461952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627012.386205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828461952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627012.386214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627012.386217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627012.386221:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828461952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627012.386259:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627012.386279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627012.386289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627012/real 1778627012] req@ffff954342f48a80 x1865013828462080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627012.386301:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828462080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627012.386309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627012.386313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627012.386317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627012.386330:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627012.386351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627012.386361:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627012/real 1778627012] req@ffff954342f4b100 x1865013828462208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627012.386373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828462208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627012.386382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627012.386386:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627012.386389:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627012.386395:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627012.386413:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627012.386422:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627012/real 1778627012] req@ffff954342f49f80 x1865013828462336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627012.386433:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828462336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627012.386442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627012.386446:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627012.386449:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627012.577811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627012.577877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627013.345780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627013.601789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627013.601835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627014.369794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627014.625819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627014.625883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627015.393743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627015.649789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627015.649851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627016.417752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627016.673764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627016.673819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627017.441737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627017.505809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627017.505825:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627017.505830:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627017.505841:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11352 00000100:00080000:1.0:1778627017.505855:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627017.505862:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627017.505907:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828462464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627017.505933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627017.505936:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627017.505938:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627017.505941:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11352 00000100:00080000:1.0:1778627017.505944:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627017.505946:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627017.505954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627017.505964:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828462592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627017.505992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627017.505996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627017.505998:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627017.506000:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828462464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627017.506004:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11352 00000100:00080000:1.0:1778627017.506011:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627017.506016:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627017.506022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627017.506037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828462720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627017.506049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627017.506053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627017.506055:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627017.506060:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11352 00000100:00080000:1.0:1778627017.506064:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627017.506068:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627017.506073:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627017.506083:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828462848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627017.506090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627017.506095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1778627017.506096:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627017.506097:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627017.506103:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11352 00000100:00080000:1.0:1778627017.506111:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627017.506115:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627017.506120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627017.506129:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828462976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627017.506154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627017/real 1778627017] req@ffff954350197800 x1865013828462464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627017.506181:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828462464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627017.506194:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627017.506200:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627017.506204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828462464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627017.506235:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627017.506261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627017.506274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627017/real 1778627017] req@ffff954350194700 x1865013828462592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627017.506288:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828462592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627017.506297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627017.506302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627017.506305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627017.506320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627017.506344:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627017.506355:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627017/real 1778627017] req@ffff954350197480 x1865013828462720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627017.506368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828462720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627017.506376:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627017.506379:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627017.506382:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627017.506389:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627017.506412:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627017.506423:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627017/real 1778627017] req@ffff954350196d80 x1865013828462848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627017.506437:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828462848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627017.506446:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627017.506450:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627017.506453:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627017.506473:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627017.506496:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627017.506508:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627017/real 1778627017] req@ffff954350194000 x1865013828462976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627017.506531:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828462976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627017.506541:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627017.506545:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627017.506548:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828462976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627017.697766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627017.697823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627018.465853:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627018.721774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627018.721846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627019.489786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627019.745746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627019.745785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627020.513762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627020.769757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627020.769828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627021.537814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627021.793752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627021.793799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627022.561790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627022.625759:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627022.625767:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627022.625771:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627022.625779:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11357 00000100:00080000:1.0:1778627022.625788:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627022.625793:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627022.625831:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828463104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627022.625858:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627022.625862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627022.625863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627022.625866:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11357 00000100:00080000:1.0:1778627022.625870:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627022.625873:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627022.625881:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627022.625898:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828463104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627022.625899:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828463232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627022.625915:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627022.625933:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627022.625935:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627022.625940:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11357 00000100:00080000:1.0:1778627022.625946:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627022.625953:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627022.625960:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627022.625971:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627022.625982:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627022/real 1778627022] req@ffff954350197100 x1865013828463104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627022.625988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828463360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627022.626003:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828463104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627022.626011:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627022.626017:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627022.626019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627022.626019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627022.626028:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11357 00000100:00080000:2.0:1778627022.626029:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627022.626034:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627022.626035:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828463104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627022.626039:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627022.626043:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627022.626056:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828463488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627022.626065:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627022.626068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463232:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627022.626069:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627022.626073:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627022.626077:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11357 00000100:00080000:1.0:1778627022.626080:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627022.626082:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627022.626092:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627022.626095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627022.626107:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828463616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627022.626108:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627022/real 1778627022] req@ffff954350194000 x1865013828463232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627022.626143:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828463232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627022.626150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627022.626153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627022.626156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627022.626161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627022.626172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627022.626177:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627022/real 1778627022] req@ffff954350196d80 x1865013828463360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627022.626184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828463360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627022.626189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627022.626191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627022.626193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627022.626202:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627022.626213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627022.626218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627022/real 1778627022] req@ffff954350197480 x1865013828463488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627022.626233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828463488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627022.626238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627022.626240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627022.626242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627022.626246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627022.626264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627022.626270:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627022/real 1778627022] req@ffff954350194700 x1865013828463616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627022.626277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828463616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627022.626282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627022.626284:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627022.626285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627022.817758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627022.817824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627023.585784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627023.841753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627023.841807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627024.609749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627024.865752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627024.865823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627025.633762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627025.889744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627025.889794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627026.657739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627026.913744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627026.913805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627027.681750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627027.745765:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627027.745774:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627027.745778:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627027.745788:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11362 00000100:00080000:0.0:1778627027.745800:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627027.745807:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627027.745861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828463744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627027.745885:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627027.745888:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627027.745891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627027.745896:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11362 00000100:00080000:0.0:1778627027.745900:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627027.745905:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627027.745914:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627027.745928:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828463872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627027.745929:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828463744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627027.745938:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627027.745942:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627027.745943:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627027.745947:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11362 00000100:00080000:0.0:1778627027.745950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627027.745969:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627027.745975:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627027.745986:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828464000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627027.745986:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627027.745994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627027.745997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627027.745999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627027.746002:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11362 00000100:00080000:0.0:1778627027.746006:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627027.746010:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627027.746014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627027.746019:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627027/real 1778627027] req@ffff954342f4a680 x1865013828463744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627027.746023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828464128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627027.746032:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627027.746035:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627027.746037:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627027.746039:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828463744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627027.746040:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11362 00000100:00080000:0.0:1778627027.746044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627027.746047:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627027.746050:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778627027.746051:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627027.746055:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627027.746059:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828464256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627027.746059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828463744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627027.746083:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627027.746106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627027.746117:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627027/real 1778627027] req@ffff954342f48700 x1865013828463872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627027.746126:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828463872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627027.746132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627027.746134:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627027.746136:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828463872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627027.746141:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627027.746153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627027.746159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627027/real 1778627027] req@ffff954342f49f80 x1865013828464000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627027.746173:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828464000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627027.746179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627027.746181:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627027.746183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627027.746186:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627027.746199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627027.746205:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627027/real 1778627027] req@ffff954342f4b100 x1865013828464128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627027.746213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828464128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627027.746218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627027.746220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627027.746221:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627027.746224:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627027.746233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627027.746238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627027/real 1778627027] req@ffff954342f48a80 x1865013828464256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627027.746245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828464256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627027.746275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627027.746278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627027.746280:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627027.937758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627027.937849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627028.705743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627028.961759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627028.961824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627029.729803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627029.985815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627029.985924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627030.753791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627031.009834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627031.009913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627031.777855:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627032.033798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627032.033865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627032.801790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627032.865800:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627032.865808:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627032.865811:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627032.865818:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11367 00000100:00080000:1.0:1778627032.865828:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627032.865834:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627032.865866:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828464384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627032.865889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627032.865892:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627032.865893:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627032.865897:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11367 00000100:00080000:1.0:1778627032.865899:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627032.865901:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627032.865907:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627032.865916:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828464512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627032.865924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627032.865926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627032.865928:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627032.865930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11367 00000100:00080000:1.0:1778627032.865944:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627032.865948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627032.865952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627032.865965:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828464640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627032.865968:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828464384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627032.865979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627032.865985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627032.865986:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627032.865989:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11367 00000100:00080000:1.0:1778627032.865991:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627032.865993:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627032.865995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627032.866001:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828464768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627032.866007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627032.866009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627032.866011:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627032.866013:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11367 00000100:00080000:1.0:1778627032.866016:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627032.866019:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627032.866022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627032.866030:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828464896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627032.866060:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627032.866104:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627032/real 1778627032] req@ffff954350197800 x1865013828464384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627032.866128:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828464384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627032.866142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627032.866147:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627032.866152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828464384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627032.866176:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627032.866213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627032.866237:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627032/real 1778627032] req@ffff954350194380 x1865013828464512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627032.866246:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828464512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627032.866253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627032.866255:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627032.866258:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627032.866285:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627032.866299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627032.866305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627032/real 1778627032] req@ffff954350195500 x1865013828464640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627032.866314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828464640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627032.866319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627032.866322:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627032.866324:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627032.866328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627032.866338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627032.866343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627032/real 1778627032] req@ffff954350195c00 x1865013828464768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627032.866351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828464768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627032.866362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627032.866364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627032.866366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627032.866371:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627032.866385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627032.866391:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627032/real 1778627032] req@ffff954350194a80 x1865013828464896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627032.866399:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828464896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627032.866405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627032.866406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627032.866408:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828464896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627033.057770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627033.057801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627033.825779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627034.081809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627034.081912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627034.849731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627035.105753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627035.105809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627035.873723:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627036.129761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627036.129831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627036.897742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627037.153748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627037.153808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627037.921735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627037.985774:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627037.985783:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627037.985802:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627037.985816:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11372 00000100:00080000:2.0:1778627037.985825:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627037.985832:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627037.985859:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828465024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627037.985881:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627037.985884:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627037.985886:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627037.985889:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11372 00000100:00080000:2.0:1778627037.985891:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627037.985893:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627037.985901:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627037.985909:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828465152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627037.985917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627037.985921:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627037.985923:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627037.985924:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828465024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627037.985927:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11372 00000100:00080000:2.0:1778627037.985929:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627037.985931:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627037.985938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627037.985948:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828465280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627037.985968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627037.985971:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627037.985972:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627037.985975:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11372 00000100:00080000:2.0:1778627037.985979:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627037.985981:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627037.985986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627037.985995:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828465408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627037.986003:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627037.986007:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627037.986008:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627037.986012:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11372 00000100:00080000:2.0:1778627037.986018:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627037.986023:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627037.986028:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627037.986036:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627037.986042:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828465536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627037.986086:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627037.986109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627037.986130:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627038.177766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627038.177822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627038.945743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627039.201752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627039.201797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627039.969766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627040.225783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627040.225869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627040.993845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627041.058011:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627041.058075:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627041.058095:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627041.058148:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627041.058159:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627041.058170:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627041.058181:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627041.058212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627037/real 1778627041] req@ffff954358788a80 x1865013828465536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627041.058246:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828465536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627041.058260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627041.058267:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627041.058272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627041.058281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627037/real 1778627041] req@ffff954358789180 x1865013828465152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627041.058294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828465152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627041.058310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627041.058314:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627041.058318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627041.058327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627037/real 1778627041] req@ffff954358788e00 x1865013828465280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627041.058371:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828465280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627041.058379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627041.058383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627041.058386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627041.058393:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627037/real 1778627041] req@ffff954358789500 x1865013828465408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627041.058405:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828465408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627041.058413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627041.058417:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627041.058420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627041.058427:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627037/real 1778627041] req@ffff954358789c00 x1865013828465024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627041.058440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828465024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627041.058448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627041.058451:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627041.058454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828465024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627041.249763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627041.249848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627042.017777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627042.273816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627042.273912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627043.041783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627043.105792:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627043.105801:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627043.105805:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627043.105814:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11377 00000100:00080000:3.0:1778627043.105824:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627043.105830:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627043.105861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828465664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627043.105885:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627043.105888:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627043.105902:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627043.105905:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11377 00000100:00080000:3.0:1778627043.105908:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627043.105911:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627043.105920:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627043.105932:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828465792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627043.105943:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627043.105946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627043.105948:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627043.105953:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11377 00000100:00080000:3.0:1778627043.105973:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627043.105979:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627043.105984:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627043.106003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828465920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627043.106008:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828465664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627043.106017:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627043.106020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627043.106021:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627043.106026:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11377 00000100:00080000:3.0:1778627043.106041:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627043.106046:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627043.106053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627043.106084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828466048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627043.106096:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627043.106121:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627043.106124:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627043.106131:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627043.106134:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11377 00000100:00080000:3.0:1778627043.106145:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627043.106150:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627043.106158:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627043.106177:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828466176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627043.106211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627043/real 1778627043] req@ffff95434701ca80 x1865013828465664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627043.106239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828465664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627043.106254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627043.106260:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627043.106265:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828465664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627043.106273:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627043.106304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627043.106320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627043/real 1778627043] req@ffff95434701e680 x1865013828465792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627043.106333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828465792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627043.106342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627043.106347:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627043.106350:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627043.106386:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627043.106410:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627043.106430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627043/real 1778627043] req@ffff95434701f480 x1865013828465920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627043.106460:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828465920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627043.106468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627043.106472:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627043.106475:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828465920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627043.106483:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627043.106506:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627043.106520:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627043/real 1778627043] req@ffff95434701df80 x1865013828466048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627043.106532:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828466048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627043.106540:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627043.106544:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627043.106547:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627043.106555:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627043.106576:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627043.106587:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627043/real 1778627043] req@ffff95434701d180 x1865013828466176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627043.106600:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828466176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627043.106626:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627043.106631:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627043.106635:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627043.297724:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627043.297857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627044.065774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627044.321763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627044.321859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627045.089774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627045.345783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627045.345853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627046.113809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627046.369784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627046.369831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627047.137862:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627047.393787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627047.393858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627048.161780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627048.225796:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627048.225805:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627048.225809:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627048.225817:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11382 00000100:00080000:1.0:1778627048.225832:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627048.225838:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627048.225888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828466304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627048.225915:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627048.225919:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627048.225921:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627048.225925:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11382 00000100:00080000:1.0:1778627048.225929:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627048.225934:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627048.225944:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627048.225964:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828466432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627048.225977:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627048.225980:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627048.225982:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627048.225991:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11382 00000100:00100000:0.0:1778627048.225993:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828466304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627048.225996:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627048.226010:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627048.226032:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627048.226046:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828466560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627048.226053:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627048.226056:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627048.226057:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627048.226060:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11382 00000100:00080000:1.0:1778627048.226071:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627048.226074:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627048.226078:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627048.226087:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627048.226093:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828466688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627048.226100:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627048.226102:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627048.226103:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627048.226107:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11382 00000100:00080000:1.0:1778627048.226109:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627048.226113:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627048.226116:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627048.226124:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828466816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627048.226156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627048/real 1778627048] req@ffff954350194e00 x1865013828466304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627048.226175:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828466304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627048.226184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627048.226188:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627048.226191:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828466304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627048.226217:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627048.226237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627048.226251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627048/real 1778627048] req@ffff954350195880 x1865013828466432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627048.226264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828466432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627048.226274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627048.226278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627048.226281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627048.226302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627048.226320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627048.226328:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627048/real 1778627048] req@ffff954350197b80 x1865013828466560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627048.226336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828466560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627048.226342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627048.226344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627048.226347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627048.226351:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627048.226362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627048.226367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627048/real 1778627048] req@ffff954350195180 x1865013828466688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627048.226375:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828466688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627048.226381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627048.226383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627048.226384:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627048.226389:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627048.226399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627048.226404:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627048/real 1778627048] req@ffff954350196a00 x1865013828466816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627048.226419:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828466816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627048.226426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627048.226428:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627048.226430:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828466816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627048.417737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627048.417825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627049.185800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627049.441852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627049.441946:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627050.209745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627050.465752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627050.465848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627051.233761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627051.489765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627051.489876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627052.257743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627052.513746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627052.513778:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627053.281775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627053.345831:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627053.345839:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627053.345842:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627053.345850:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11387 00000100:00080000:3.0:1778627053.345860:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627053.345866:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627053.345899:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828466944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627053.345935:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627053.345938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627053.345939:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627053.345942:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11387 00000100:00080000:3.0:1778627053.345946:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627053.345948:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627053.345955:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627053.345965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828467072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627053.345972:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627053.345974:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627053.345975:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627053.345978:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11387 00000100:00080000:3.0:1778627053.345980:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627053.345982:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627053.345986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627053.345989:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828466944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627053.345994:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828467200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627053.346004:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627053.346006:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627053.346007:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627053.346010:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11387 00000100:00080000:3.0:1778627053.346017:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627053.346020:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627053.346024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627053.346034:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828467328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627053.346040:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627053.346042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627053.346043:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627053.346045:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11387 00000100:00080000:3.0:1778627053.346047:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627053.346050:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627053.346052:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627053.346058:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828467456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627053.346066:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627053.346111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627053/real 1778627053] req@ffff95434701f800 x1865013828466944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627053.346132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828466944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627053.346143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627053.346148:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627053.346152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828466944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627053.346160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627053.346182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627053.346193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627053/real 1778627053] req@ffff95434701d880 x1865013828467072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627053.346206:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828467072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627053.346221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627053.346225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627053.346228:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627053.346253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627053.346273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627053.346284:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627053/real 1778627053] req@ffff95434701ea00 x1865013828467200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627053.346297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828467200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627053.346305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627053.346319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627053.346323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627053.346330:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627053.346350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627053.346360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627053/real 1778627053] req@ffff95434701c000 x1865013828467328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627053.346372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828467328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627053.346380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627053.346382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627053.346401:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627053.346409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627053.346431:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627053.346443:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627053/real 1778627053] req@ffff95434701c700 x1865013828467456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627053.346456:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828467456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627053.346464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627053.346467:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627053.346470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627053.537761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627053.537850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627054.305753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627054.561772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627054.561821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627055.329815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627055.585824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627055.585863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627056.353756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627056.609812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627056.609814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778627057.377757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627057.633790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627057.633819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627058.401726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627058.465777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627058.465786:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627058.465808:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627058.465818:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11392 00000100:00080000:1.0:1778627058.465829:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627058.465837:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627058.465869:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828467584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627058.465894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627058.465898:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627058.465900:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627058.465905:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11392 00000100:00080000:1.0:1778627058.465910:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627058.465924:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627058.465930:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828467584:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627058.465937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627058.465957:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828467712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627058.465969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627058.465973:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627058.465975:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627058.465979:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11392 00000100:00080000:1.0:1778627058.465984:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627058.465988:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627058.465993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778627058.465997:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627058.466008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828467840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627058.466032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627058.466036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1778627058.466038:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627058/real 1778627058] req@ffff954350195f80 x1865013828467584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627058.466039:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627058.466050:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11392 00000100:00080000:1.0:1778627058.466056:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627058.466061:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828467584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627058.466062:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627058.466072:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:0.0:1778627058.466074:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627058.466080:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627058.466090:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828467968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627058.466093:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828467584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627058.466108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627058.466112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627058.466114:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627058.466119:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11392 00000100:00080000:1.0:1778627058.466123:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627058.466125:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467712:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627058.466131:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627058.466136:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627058.466149:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828468096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627058.466154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627058.466167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627058/real 1778627058] req@ffff954350196a00 x1865013828467712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627058.466177:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828467712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627058.466184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627058.466187:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627058.466190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627058.466194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627058.466206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627058.466211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627058/real 1778627058] req@ffff954350195180 x1865013828467840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627058.466218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828467840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627058.466223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627058.466225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627058.466227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627058.466230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627058.466240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627058.466252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627058/real 1778627058] req@ffff954350197b80 x1865013828467968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627058.466259:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828467968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627058.466264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627058.466266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627058.466276:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828467968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627058.466287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627058.466299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627058.466305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627058/real 1778627058] req@ffff954350195880 x1865013828468096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627058.466312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828468096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627058.466318:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627058.466320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627058.466322:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627058.657746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627058.657827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627059.425854:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627059.681799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627059.681854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627060.449748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627060.705763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627060.705800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627061.473753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627061.729752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627061.729795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627062.497774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627062.753740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627062.753765:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627063.521739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627063.585737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627063.585746:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627063.585751:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627063.585760:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11398 00000100:00080000:1.0:1778627063.585785:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627063.585793:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627063.585826:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828468224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627063.585848:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627063.585851:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627063.585852:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627063.585855:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11398 00000100:00080000:1.0:1778627063.585858:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627063.585861:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627063.585867:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627063.585877:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828468352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627063.585885:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627063.585887:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627063.585889:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627063.585894:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828468224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627063.585894:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11398 00000100:00080000:1.0:1778627063.585899:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627063.585903:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627063.585906:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627063.585913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828468480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627063.585919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627063.585926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627063.585928:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627063.585931:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11398 00000100:00080000:1.0:1778627063.585943:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627063.585946:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627063.585950:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627063.585953:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627063.585961:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828468608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627063.585969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627063.585971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627063.585973:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627063.585975:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11398 00000100:00080000:1.0:1778627063.585978:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627063.585980:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627063.585983:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627063.585992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828468736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627063.585996:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627063/real 1778627063] req@ffff954350194e00 x1865013828468224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627063.586015:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828468224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627063.586024:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627063.586027:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627063.586030:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828468224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627063.586048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627063.586068:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627063.586076:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627063/real 1778627063] req@ffff954350194a80 x1865013828468352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627063.586084:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828468352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627063.586089:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627063.586091:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627063.586094:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627063.586101:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627063.586111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627063.586117:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627063/real 1778627063] req@ffff954350195c00 x1865013828468480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627063.586124:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828468480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627063.586138:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627063.586140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627063.586142:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627063.586146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627063.586158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627063.586164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627063/real 1778627063] req@ffff954350195500 x1865013828468608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627063.586171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828468608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627063.586176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627063.586178:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627063.586179:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627063.586183:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627063.586192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627063.586197:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627063/real 1778627063] req@ffff954350194380 x1865013828468736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627063.586204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828468736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627063.586209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627063.586210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627063.586217:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627063.777767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627063.777811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627064.545765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627064.801733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627064.801787:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627065.569771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627065.825744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627065.825766:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627066.593786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627066.849780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627066.849820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627067.617746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627067.873747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627067.873826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627068.641786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627068.705764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627068.705771:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627068.705775:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627068.705783:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11403 00000100:00080000:0.0:1778627068.705793:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627068.705799:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627068.705830:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828468864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627068.705854:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627068.705858:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627068.705860:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627068.705864:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11403 00000100:00080000:0.0:1778627068.705868:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627068.705872:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627068.705882:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627068.705887:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828468864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627068.705898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828468992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627068.705912:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627068.705914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627068.705916:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627068.705932:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11403 00000100:00080000:0.0:1778627068.705936:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627068.705941:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627068.705942:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627068.705949:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627068.705959:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828469120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627068.705967:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627068.705969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627068.705971:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627068.705975:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11403 00000100:00100000:2.0:1778627068.705975:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627068/real 1778627068] req@ffff954342f4ad80 x1865013828468864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627068.705977:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627068.705983:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627068.705990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627068.705999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828469248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627068.706003:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828468864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627068.706014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627068.706021:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627068.706023:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627068.706029:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11403 00000100:00080000:2.0:1778627068.706032:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627068.706033:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627068.706035:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627068.706037:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627068.706044:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828469376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627068.706049:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627068.706057:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828468864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627068.706093:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627068.706118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627068.706131:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627068/real 1778627068] req@ffff954342f49500 x1865013828468992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627068.706141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828468992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627068.706148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627068.706150:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627068.706152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828468992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627068.706156:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627068.706172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627068.706178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627068/real 1778627068] req@ffff954342f4bb80 x1865013828469120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627068.706186:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828469120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627068.706191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627068.706193:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627068.706194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627068.706198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627068.706217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627068.706223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627068/real 1778627068] req@ffff954342f4b480 x1865013828469248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627068.706230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828469248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627068.706236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627068.706237:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627068.706239:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627068.706243:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627068.706252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627068.706257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627068/real 1778627068] req@ffff954342f4a300 x1865013828469376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627068.706264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828469376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627068.706270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627068.706273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627068.706276:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627068.897774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627068.897836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627069.665758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778627069.921951:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627069.922322:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778627070.689750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627070.945762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627070.945947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627071.713775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627071.969764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627071.969841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627072.737757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627072.993913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627072.994086:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778627073.761750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627073.825829:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627073.825840:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627073.825869:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627073.825879:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11408 00000100:00080000:1.0:1778627073.825891:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627073.825898:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627073.825937:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828469504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627073.825969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627073.825972:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627073.825974:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627073.825977:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11408 00000100:00080000:1.0:1778627073.825979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627073.825982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627073.825991:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627073.826001:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828469632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627073.826008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627073.826011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627073.826012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627073.826016:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11408 00000100:00080000:1.0:1778627073.826018:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627073.826020:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627073.826023:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627073.826031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828469760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627073.826034:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828469504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627073.826059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627073.826062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627073.826064:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627073.826070:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11408 00000100:00080000:1.0:1778627073.826075:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627073.826081:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627073.826087:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627073.826102:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828469888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627073.826112:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627073.826117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627073.826119:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778627073.826130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627073.826134:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11408 00000100:00080000:1.0:1778627073.826140:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627073.826144:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627073.826149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627073.826162:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828470016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627073.826179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627073/real 1778627073] req@ffff954350197800 x1865013828469504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627073.826205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828469504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627073.826217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627073.826222:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627073.826226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828469504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627073.826235:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627073.826262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627073.826275:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627073/real 1778627073] req@ffff954350194700 x1865013828469632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627073.826288:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828469632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627073.826298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627073.826302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627073.826305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627073.826332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627073.826378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627073.826389:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627073/real 1778627073] req@ffff954350197480 x1865013828469760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627073.826402:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828469760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627073.826410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627073.826413:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627073.826417:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627073.826423:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627073.826440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627073.826447:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627073/real 1778627073] req@ffff954350196d80 x1865013828469888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627073.826454:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828469888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627073.826459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627073.826461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627073.826463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828469888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627073.826467:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627073.826477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627073.826482:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627073/real 1778627073] req@ffff954350194000 x1865013828470016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627073.826499:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828470016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627073.826505:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627073.826507:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627073.826509:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627074.018004:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627074.018110:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627074.785782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627075.041779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627075.041820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627075.809745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627076.065740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627076.065776:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627076.833744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627077.089773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627077.089808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627077.857735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627078.113741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627078.113804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627078.881765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627078.945772:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627078.945783:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627078.945787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627078.945797:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11413 00000100:00080000:1.0:1778627078.945825:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627078.945832:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627078.945864:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828470144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627078.945893:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627078.945897:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627078.945899:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627078.945903:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11413 00000100:00080000:1.0:1778627078.945907:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627078.945912:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627078.945922:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627078.945934:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828470144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627078.945940:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828470272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627078.945950:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627078.945952:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627078.945953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627078.945955:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11413 00000100:00080000:1.0:1778627078.945959:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627078.945962:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627078.945965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627078.945976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828470400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627078.945983:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627078.946001:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627078.946003:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627078.946007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11413 00000100:00080000:1.0:1778627078.946010:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627078.946013:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627078.946019:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627078.946020:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627078.946032:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828470528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627078.946039:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627078.946042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627078.946043:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627078.946061:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627078/real 1778627078] req@ffff954350197100 x1865013828470144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627078.946065:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11413 00000100:00080000:1.0:1778627078.946069:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627078.946071:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627078.946077:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627078.946088:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828470656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627078.946093:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828470144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627078.946108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627078.946112:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627078.946135:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828470144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627078.946161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627078.946186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627078.946196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627078/real 1778627078] req@ffff954350194000 x1865013828470272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627078.946205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828470272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627078.946211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627078.946213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627078.946215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627078.946219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627078.946241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627078.946261:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627078/real 1778627078] req@ffff954350196d80 x1865013828470400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627078.946271:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828470400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627078.946276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627078.946278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627078.946280:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627078.946284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627078.946298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627078.946311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627078/real 1778627078] req@ffff954350197480 x1865013828470528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627078.946319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828470528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627078.946325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627078.946327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627078.946329:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627078.946332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627078.946343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627078.946348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627078/real 1778627078] req@ffff954350194700 x1865013828470656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627078.946355:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828470656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627078.946361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627078.946363:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627078.946364:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627079.137797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627079.137852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627079.905739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627080.161759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627080.161841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627080.929758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627081.185786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627081.185844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627081.953778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627082.209776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627082.209857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627082.977782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627083.233829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627083.233894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627084.001780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627084.065800:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627084.065814:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627084.065821:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627084.065833:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11418 00000100:00080000:3.0:1778627084.065847:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627084.065855:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627084.065904:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828470784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627084.065957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627084.065961:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627084.065964:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627084.065969:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11418 00000100:00080000:3.0:1778627084.065974:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627084.065979:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627084.065991:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627084.066008:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828470912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627084.066022:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627084.066026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627084.066029:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627084.066037:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11418 00000100:00100000:2.0:1778627084.066039:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828470784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627084.066042:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627084.066053:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627084.066059:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627084.066074:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828471040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627084.066085:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627084.066089:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627084.066091:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627084.066097:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11418 00000800:00000100:2.0:1778627084.066112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627084.066114:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627084.066123:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627084.066130:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627084.066163:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627084/real 1778627084] req@ffff95434701f100 x1865013828470784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627084.066164:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828471168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627084.066208:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627084.066212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828470784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627084.066216:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627084.066224:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627084.066234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778627084.066238:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11418 00000100:00080000:2.0:1778627084.066243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627084.066251:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828470784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627084.066251:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627084.066264:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627084.066271:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627084.066281:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778627084.066292:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828471296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627084.066306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627084.066323:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627084/real 1778627084] req@ffff95434701c380 x1865013828470912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627084.066333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828470912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627084.066340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627084.066344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627084.066348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828470912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627084.066355:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627084.066402:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627084.066413:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627084/real 1778627084] req@ffff95434701d500 x1865013828471040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627084.066421:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828471040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627084.066427:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627084.066429:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627084.066431:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627084.066436:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627084.066449:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627084.066455:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627084/real 1778627084] req@ffff95434701c700 x1865013828471168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627084.066465:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828471168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627084.066474:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627084.066477:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627084.066480:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627084.066498:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627084.066519:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627084.066528:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627084/real 1778627084] req@ffff95434701c000 x1865013828471296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627084.066547:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828471296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627084.066552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627084.066555:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627084.066557:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627084.257807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627084.257898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627085.025787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627085.281807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627085.281916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627086.049787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627086.305835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627086.305919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627087.073756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627087.329769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627087.329834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627088.097774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627088.353791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627088.353891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627089.121737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627089.185773:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627089.185784:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627089.185788:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627089.185799:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11423 00000100:00080000:3.0:1778627089.185812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627089.185820:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627089.185883:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828471424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627089.185911:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627089.185914:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627089.185916:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627089.185919:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11423 00000100:00080000:3.0:1778627089.185922:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627089.185925:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627089.185934:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627089.185943:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828471552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627089.185951:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627089.185954:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627089.185956:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627089.185958:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11423 00000100:00080000:3.0:1778627089.185961:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627089.185963:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627089.185967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627089.185975:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828471680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627089.185982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627089.185988:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627089.185989:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627089.185993:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828471424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627089.186001:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11423 00000100:00080000:3.0:1778627089.186010:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627089.186013:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627089.186018:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627089.186026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828471808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627089.186033:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627089.186036:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627089.186038:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627089.186040:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11423 00000100:00080000:3.0:1778627089.186042:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627089.186062:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627089.186066:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627089.186075:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828471936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627089.186088:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627089.186134:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627089/real 1778627089] req@ffff95434701ea00 x1865013828471424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627089.186148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828471424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627089.186156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627089.186160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627089.186162:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828471424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627089.186182:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627089.186201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627089.186209:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627089/real 1778627089] req@ffff95434701d880 x1865013828471552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627089.186217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828471552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627089.186222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627089.186225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627089.186227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627089.186250:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627089.186270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627089.186281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627089/real 1778627089] req@ffff95434701f800 x1865013828471680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627089.186294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828471680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627089.186302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627089.186305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627089.186307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627089.186312:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627089.186327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627089.186334:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627089/real 1778627089] req@ffff95434701d180 x1865013828471808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627089.186348:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828471808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627089.186354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627089.186356:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627089.186358:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627089.186362:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627089.186377:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627089.186387:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627089/real 1778627089] req@ffff95434701df80 x1865013828471936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627089.186398:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828471936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627089.186408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627089.186411:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627089.186415:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828471936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627089.377784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627089.377929:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627090.145807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627090.401799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627090.401909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627091.169776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627091.425766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627091.425915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627092.193844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627092.449826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627092.449911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627093.217781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627093.473812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627093.473894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627094.241823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627094.305775:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627094.305785:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627094.305789:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627094.305798:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11428 00000100:00080000:3.0:1778627094.305808:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627094.305813:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627094.305877:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828472064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627094.305904:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627094.305907:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627094.305908:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627094.305912:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11428 00000100:00080000:3.0:1778627094.305915:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627094.305920:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627094.305929:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627094.305953:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828472192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627094.305966:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627094.305969:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627094.305971:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627094.305976:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11428 00000100:00080000:3.0:1778627094.305981:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627094.305985:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627094.305993:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627094.306006:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828472320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627094.306018:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627094.306022:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627094.306027:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627094.306029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828472064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627094.306048:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11428 00000100:00080000:3.0:1778627094.306052:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627094.306055:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627094.306060:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627094.306072:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828472448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627094.306079:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627094.306081:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627094.306083:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627094.306085:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11428 00000100:00080000:3.0:1778627094.306088:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627094.306090:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627094.306093:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627094.306101:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828472576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627094.306138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627094.306187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627094/real 1778627094] req@ffff95434701f480 x1865013828472064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627094.306208:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828472064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627094.306219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627094.306223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627094.306227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828472064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627094.306234:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627094.306255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627094.306267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627094/real 1778627094] req@ffff95434701e680 x1865013828472192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627094.306281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828472192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627094.306292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627094.306296:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627094.306301:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627094.306335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627094.306356:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627094.306365:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627094/real 1778627094] req@ffff95434701ca80 x1865013828472320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627094.306390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828472320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627094.306396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627094.306399:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627094.306402:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627094.306407:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627094.306424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627094.306431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627094/real 1778627094] req@ffff95434701df80 x1865013828472448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627094.306440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828472448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627094.306447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627094.306449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627094.306451:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627094.306456:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627094.306467:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627094.306473:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627094/real 1778627094] req@ffff95434701d180 x1865013828472576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627094.306481:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828472576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627094.306498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627094.306501:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627094.306503:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627094.497804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627094.497889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627095.265762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627095.521777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627095.521848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627096.289848:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627096.545808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627096.545878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627097.313792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627097.569788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627097.569845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627098.337799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627098.593807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627098.593903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627099.361820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627099.425870:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627099.425901:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627099.425906:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627099.425920:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11433 00000100:00080000:0.0:1778627099.425934:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627099.425943:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627099.425995:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828472704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627099.426027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627099.426032:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627099.426034:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627099.426039:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11433 00000100:00080000:0.0:1778627099.426043:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627099.426049:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627099.426061:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627099.426074:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828472704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627099.426080:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828472832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627099.426094:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627099.426098:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627099.426119:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627099.426124:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11433 00000100:00080000:0.0:1778627099.426130:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627099.426135:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627099.426142:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778627099.426149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627099.426158:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828472960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627099.426171:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627099.426176:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627099.426178:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627099.426184:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11433 00000100:00080000:0.0:1778627099.426188:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627099.426192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627099/real 1778627099] req@ffff954342f49880 x1865013828472704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627099.426196:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627099.426207:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627099.426216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828472704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627099.426224:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828473088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627099.426229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627099.426237:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627099.426242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828472704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627099.426243:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627099.426249:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627099.426252:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627099.426257:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11433 00000100:00080000:0.0:1778627099.426261:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627099.426266:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627099.426270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472832:192.168.204.102@tcp:38:kworker.0 00000008:00100000:0.0:1778627099.426274:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627099.426288:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627099.426300:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828473216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627099.426309:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627099/real 1778627099] req@ffff954342f49c00 x1865013828472832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627099.426323:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828472832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627099.426330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627099.426333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627099.426335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627099.426340:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627099.426355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627099.426360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627099/real 1778627099] req@ffff954342f49180 x1865013828472960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627099.426368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828472960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627099.426373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627099.426375:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627099.426376:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828472960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627099.426380:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627099.426391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627099.426396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627099/real 1778627099] req@ffff954342f48000 x1865013828473088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627099.426403:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828473088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627099.426416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627099.426418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627099.426420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627099.426432:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627099.426445:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627099.426451:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627099/real 1778627099] req@ffff954342f48380 x1865013828473216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627099.426458:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828473216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627099.426464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627099.426466:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627099.426468:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627099.617736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627099.617821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627100.385803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627100.641793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627100.641882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627101.409834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627101.665814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627101.665900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627102.433760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627102.689800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627102.689869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627103.457820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627103.713792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627103.713893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627104.481778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627104.545885:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627104.545895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627104.545900:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627104.545927:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11439 00000100:00080000:3.0:1778627104.545940:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627104.545947:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627104.546015:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828473344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627104.546052:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627104.546087:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627104.546090:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627104.546094:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11439 00000100:00080000:3.0:1778627104.546106:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627104.546113:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627104.546122:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627104.546129:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828473344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627104.546140:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828473472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627104.546152:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627104.546165:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627104.546166:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627104.546169:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11439 00000100:00080000:3.0:1778627104.546171:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627104.546182:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627104.546185:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627104.546197:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828473600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627104.546205:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627104.546207:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627104.546209:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627104.546211:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11439 00000100:00080000:3.0:1778627104.546213:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627104.546217:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627104.546229:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627104.546237:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828473728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627104.546244:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627104.546247:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627104.546248:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627104.546251:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11439 00000100:00080000:3.0:1778627104.546254:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627104.546257:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627104.546260:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627104.546267:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828473856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627104.546330:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627104.546394:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627104.546417:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627104.546450:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627104.737779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627104.737843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627105.505838:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627105.761801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627105.761871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627106.529785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627106.785769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627106.785844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627107.553808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1778627107.617903:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:0.0:1778627107.617943:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1778627107.617960:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778627107.618011:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778627107.618022:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778627107.618034:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778627107.618046:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778627107.618056:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627104/real 1778627107] req@ffff95434701d880 x1865013828473472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627107.618094:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828473472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627107.618108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627107.618114:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627107.618118:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627107.618127:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627104/real 1778627107] req@ffff95434701ea00 x1865013828473600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627107.618142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828473600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627107.618150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627107.618170:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627107.618175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627107.618183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627104/real 1778627107] req@ffff95434701c000 x1865013828473728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627107.618198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828473728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627107.618206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627107.618210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627107.618214:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627107.618221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627104/real 1778627107] req@ffff95434701c700 x1865013828473856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627107.618233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828473856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627107.618243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627107.618246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627107.618250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828473856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627107.618256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627104/real 1778627107] req@ffff95434701f800 x1865013828473344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627107.618267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828473344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627107.618275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627107.618286:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627107.618290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828473344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627107.809747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627107.809804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627108.577747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627108.833722:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627108.833797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627109.601787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627109.665829:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627109.665837:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627109.665841:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627109.665849:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11444 00000100:00080000:1.0:1778627109.665858:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627109.665864:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627109.665894:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828473984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627109.665925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627109.665929:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627109.665931:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627109.665935:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11444 00000100:00080000:1.0:1778627109.665938:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627109.665943:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627109.665959:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627109.665976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828474112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627109.665982:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828473984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627109.666018:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627109.666022:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627109.666024:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627109.666029:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11444 00000100:00080000:1.0:1778627109.666035:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627109.666039:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627109.666046:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778627109.666053:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627109.666062:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828474240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627109.666073:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627109.666076:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627109.666078:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627109.666084:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11444 00000100:00080000:1.0:1778627109.666089:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627109.666091:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627109/real 1778627109] req@ffff954350197800 x1865013828473984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627109.666097:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627109.666108:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627109.666123:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828473984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627109.666124:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828474368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627109.666150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627109.666152:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627109.666158:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627109.666158:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627109.666162:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627109.666166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828473984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627109.666171:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11444 00000100:00080000:1.0:1778627109.666184:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627109.666189:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627109.666195:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627109.666198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627109.666211:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828474496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627109.666229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627109.666244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627109/real 1778627109] req@ffff954350194380 x1865013828474112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627109.666258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828474112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627109.666267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627109.666282:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627109.666294:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627109.666302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627109.666325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627109.666336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627109/real 1778627109] req@ffff954350195500 x1865013828474240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627109.666349:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828474240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627109.666357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627109.666360:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627109.666363:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627109.666370:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627109.666388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627109.666399:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627109/real 1778627109] req@ffff954350195c00 x1865013828474368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627109.666410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828474368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627109.666418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627109.666421:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627109.666424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627109.666441:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627109.666479:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627109.666498:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627109/real 1778627109] req@ffff954350194a80 x1865013828474496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627109.666518:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828474496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627109.666528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627109.666532:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627109.666535:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627109.857763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627109.857826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627110.625740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627110.881753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627110.881807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627111.649781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627111.905777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627111.905847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627112.673735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627112.929745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627112.929816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627113.697782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627113.953784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627113.953843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627114.721735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627114.785767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627114.785778:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627114.785783:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627114.785793:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11449 00000100:00080000:1.0:1778627114.785804:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627114.785812:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627114.785851:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828474624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627114.785881:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627114.785885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627114.785887:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627114.785892:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11449 00000100:00080000:1.0:1778627114.785896:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627114.785900:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627114.785910:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627114.785922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828474624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627114.785946:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828474752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627114.785959:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627114.785962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627114.785964:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627114.785969:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11449 00000100:00080000:1.0:1778627114.785974:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627114.785981:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627114.785982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627114.785990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627114.786004:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828474880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627114.786014:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627114/real 1778627114] req@ffff954350194e00 x1865013828474624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627114.786016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627114.786027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627114.786031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627114.786038:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11449 00000100:00100000:2.0:1778627114.786042:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828474624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627114.786044:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627114.786053:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627114.786055:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778627114.786059:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:2.0:1778627114.786061:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627114.786072:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828474624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627114.786075:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828475008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627114.786087:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627114.786089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627114.786091:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627114.786098:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474752:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627114.786108:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11449 00000100:00080000:1.0:1778627114.786114:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627114.786119:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627114.786124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627114.786127:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627114.786135:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627114/real 1778627114] req@ffff954350195880 x1865013828474752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627114.786144:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828475136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627114.786148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828474752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627114.786171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627114.786175:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627114.786179:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627114.786184:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627114.786197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627114.786203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627114/real 1778627114] req@ffff954350197b80 x1865013828474880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627114.786210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828474880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627114.786215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627114.786217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627114.786219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828474880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627114.786223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627114.786233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627114.786246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627114/real 1778627114] req@ffff954350195180 x1865013828475008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627114.786254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828475008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627114.786259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627114.786261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627114.786263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627114.786274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627114.786285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627114.786290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627114/real 1778627114] req@ffff954350196a00 x1865013828475136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627114.786297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828475136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627114.786303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627114.786305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627114.786307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627114.977744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627114.977811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627115.745736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627116.001788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627116.001849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627116.769745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627117.025742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627117.025797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627117.793834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627118.049749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627118.049842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627118.817747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627119.073739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627119.073818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627119.841742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627119.905806:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627119.905817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627119.905822:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627119.905832:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11454 00000100:00080000:1.0:1778627119.905845:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627119.905852:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627119.905897:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828475264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627119.905926:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627119.905930:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627119.905932:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627119.905937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11454 00000100:00080000:1.0:1778627119.905940:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627119.905945:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627119.905977:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627119.905997:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828475392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627119.905997:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828475264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627119.906018:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627119.906020:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627119.906022:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627119.906026:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11454 00000100:00080000:1.0:1778627119.906030:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627119.906038:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627119.906041:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627119.906051:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828475520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627119.906060:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627119.906063:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627119.906065:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627119.906070:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11454 00000100:00080000:1.0:1778627119.906075:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627119.906079:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627119.906085:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627119.906096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828475648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627119.906108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778627119.906117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627119.906118:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627119.906129:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627119.906137:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11454 00000100:00080000:1.0:1778627119.906145:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627119.906151:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627119.906158:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627119.906162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627119/real 1778627119] req@ffff954350195f80 x1865013828475264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627119.906178:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828475776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627119.906184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828475264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627119.906237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627119.906241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627119.906246:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828475264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627119.906269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627119.906289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627119.906296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627119/real 1778627119] req@ffff954350196a00 x1865013828475392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627119.906305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828475392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627119.906311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627119.906313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627119.906315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627119.906319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627119.906328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627119.906333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627119/real 1778627119] req@ffff954350195180 x1865013828475520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627119.906352:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828475520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627119.906357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627119.906359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627119.906361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627119.906365:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627119.906378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627119.906383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627119/real 1778627119] req@ffff954350197b80 x1865013828475648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627119.906390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828475648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627119.906395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627119.906397:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627119.906399:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627119.906402:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627119.906412:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627119.906417:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627119/real 1778627119] req@ffff954350195880 x1865013828475776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627119.906424:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828475776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627119.906457:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627119.906461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627119.906464:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828475776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627120.097766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627120.097858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627120.865773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627121.121739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627121.121814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627121.889761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627122.145762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627122.145887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627122.913808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627123.169859:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627123.169984:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627123.937786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627124.193793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627124.194019:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627124.961826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627125.025767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627125.025778:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627125.025783:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627125.025795:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11459 00000100:00080000:1.0:1778627125.025809:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627125.025816:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627125.025862:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828475904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627125.025891:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627125.025899:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627125.025901:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627125.025905:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11459 00000100:00080000:1.0:1778627125.025908:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627125.025930:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627125.025955:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627125.025967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828476032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627125.025976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627125.025979:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627125.025981:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627125.025983:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11459 00000100:00080000:1.0:1778627125.025987:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627125.025990:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627125.025995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627125.025996:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828475904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627125.026006:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828476160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627125.026014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627125.026017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627125.026018:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627125.026022:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11459 00000100:00080000:1.0:1778627125.026025:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627125.026027:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627125.026031:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627125.026040:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828476288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627125.026047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627125.026055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627125.026056:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627125.026059:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11459 00000100:00080000:1.0:1778627125.026066:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627125.026069:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627125.026073:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627125.026079:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627125.026084:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828476416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627125.026123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627125/real 1778627125] req@ffff954350194e00 x1865013828475904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627125.026142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828475904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627125.026152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627125.026156:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627125.026159:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828475904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627125.026164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627125.026180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627125.026188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627125/real 1778627125] req@ffff954350194a80 x1865013828476032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627125.026197:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828476032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627125.026217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627125.026220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627125.026223:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627125.026248:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627125.026264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627125.026271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627125/real 1778627125] req@ffff954350195c00 x1865013828476160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627125.026279:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828476160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627125.026285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627125.026286:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627125.026288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627125.026293:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627125.026304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627125.026310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627125/real 1778627125] req@ffff954350195500 x1865013828476288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627125.026318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828476288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627125.026323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627125.026325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627125.026332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627125.026358:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627125.026376:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627125.026383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627125/real 1778627125] req@ffff954350194380 x1865013828476416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627125.026392:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828476416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627125.026399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627125.026401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627125.026403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627125.217834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627125.217987:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627125.985777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627126.241795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627126.241864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627127.009788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627127.265785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627127.265877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627128.033794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627128.289761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627128.289849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627129.057819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627129.313814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627129.313913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627130.081752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627130.145782:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627130.145794:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627130.145799:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627130.145809:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11464 00000100:00080000:1.0:1778627130.145822:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627130.145830:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627130.145879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828476544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627130.145907:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627130.145911:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627130.145913:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627130.145918:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11464 00000100:00080000:1.0:1778627130.145923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627130.145928:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627130.145938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627130.145949:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828476544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627130.145953:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828476672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627130.145983:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627130.145994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627130.145997:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627130.146002:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11464 00000800:00000100:2.0:1778627130.146007:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627130.146011:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627130.146017:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627130.146025:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627130.146042:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627130/real 1778627130] req@ffff954350197800 x1865013828476544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627130.146043:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828476800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627130.146070:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627130.146074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627130.146077:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627130.146084:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11464 00000100:00100000:2.0:1778627130.146084:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828476544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627130.146090:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627130.146101:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627130.146104:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627130.146110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:1.0:1778627130.146114:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627130.146123:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828476544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627130.146130:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828476928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627130.146141:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627130.146144:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627130.146147:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627130.146151:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476672:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627130.146155:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11464 00000100:00080000:1.0:1778627130.146170:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627130.146176:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627130.146182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627130.146183:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627130.146201:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828477056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627130.146212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627130/real 1778627130] req@ffff954350194700 x1865013828476672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627130.146230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828476672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627130.146251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627130.146256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627130.146260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627130.146268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627130.146301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627130.146313:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627130/real 1778627130] req@ffff954350197480 x1865013828476800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627130.146326:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828476800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627130.146335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627130.146338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627130.146341:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627130.146348:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627130.146371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627130.146393:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627130/real 1778627130] req@ffff954350196d80 x1865013828476928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627130.146407:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828476928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627130.146416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627130.146419:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627130.146423:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828476928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627130.146441:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627130.146462:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627130.146473:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627130/real 1778627130] req@ffff954350194000 x1865013828477056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627130.146486:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828477056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627130.146496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627130.146499:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627130.146503:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627130.337762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627130.337843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627131.105738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627131.361766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627131.361825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627132.129737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627132.385747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627132.385816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627133.153829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627133.409849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627133.409971:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627134.177770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627134.433784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627134.433803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627135.201740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627135.265777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627135.265802:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627135.265805:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627135.265813:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11469 00000100:00080000:0.0:1778627135.265822:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627135.265828:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627135.265857:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828477184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627135.265881:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627135.265883:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627135.265885:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627135.265888:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11469 00000100:00080000:0.0:1778627135.265890:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627135.265893:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627135.265900:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627135.265909:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828477312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627135.265917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627135.265920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778627135.265920:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828477184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627135.265922:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627135.265929:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11469 00000100:00080000:0.0:1778627135.265931:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627135.265933:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627135.265936:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627135.265946:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828477440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627135.265958:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627135.265960:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627135.265961:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627135.265964:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11469 00000100:00080000:0.0:1778627135.265968:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627135.265970:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627135.265974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627135.265982:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627135.265983:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828477568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627135.265990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627135.265993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627135.265995:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627135.265997:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11469 00000100:00080000:0.0:1778627135.265999:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627135.266002:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627135.266005:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627135.266011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828477696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627135.266026:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627135/real 1778627135] req@ffff954342f48e00 x1865013828477184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627135.266043:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828477184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627135.266051:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627135.266054:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627135.266056:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828477184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627135.266074:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627135.266089:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627135.266098:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627135/real 1778627135] req@ffff954342f4aa00 x1865013828477312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627135.266111:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828477312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627135.266120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627135.266124:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627135.266127:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627135.266147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627135.266166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627135.266174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627135/real 1778627135] req@ffff954342f48380 x1865013828477440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627135.266183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828477440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627135.266198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627135.266201:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627135.266203:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627135.266207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627135.266221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627135.266227:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627135/real 1778627135] req@ffff954342f48000 x1865013828477568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627135.266235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828477568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627135.266240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627135.266242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627135.266243:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627135.266247:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627135.266255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627135.266260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627135/real 1778627135] req@ffff954342f49180 x1865013828477696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627135.266266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828477696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627135.266272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627135.266273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627135.266281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627135.457770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627135.457816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627136.225740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627136.481903:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627136.481935:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627137.249770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627137.505808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627137.505924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627138.273758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627138.529822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627138.529941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627139.297778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627139.553875:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627139.554219:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627140.321802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627140.385776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627140.385789:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627140.385795:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627140.385810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11474 00000100:00080000:0.0:1778627140.385824:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627140.385832:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627140.385883:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828477824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627140.385919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627140.385923:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627140.385925:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627140.385930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11474 00000100:00080000:0.0:1778627140.385934:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627140.385939:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627140.385952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627140.385970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828477952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627140.385979:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828477824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627140.386010:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627140.386014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627140.386016:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627140.386023:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11474 00000100:00080000:0.0:1778627140.386028:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627140.386046:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627140.386053:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778627140.386090:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627140.386096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828478080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627140.386109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627140.386114:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627140.386129:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627140.386140:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11474 00000100:00080000:0.0:1778627140.386146:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627140.386154:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627140.386156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627140/real 1778627140] req@ffff954342f49c00 x1865013828477824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778627140.386166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627140.386180:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828477824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627140.386188:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828478208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627140.386193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627140.386213:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627140.386217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627140.386224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828477824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627140.386225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627140.386231:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627140.386247:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11474 00000100:00080000:0.0:1778627140.386253:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627140.386257:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477952:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778627140.386262:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627140.386271:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627140.386290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828478336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627140.386298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627140.386320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627140/real 1778627140] req@ffff954342f49880 x1865013828477952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627140.386330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828477952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627140.386340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627140.386377:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627140.386381:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828477952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627140.386389:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627140.386411:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627140.386420:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627140/real 1778627140] req@ffff954342f4a300 x1865013828478080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627140.386429:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828478080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627140.386435:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627140.386438:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627140.386442:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627140.386449:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627140.386479:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627140.386492:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627140/real 1778627140] req@ffff954342f4b480 x1865013828478208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627140.386510:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828478208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627140.386515:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627140.386518:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627140.386520:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627140.386549:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627140.386571:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627140.386579:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627140/real 1778627140] req@ffff954342f4bb80 x1865013828478336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627140.386587:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828478336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627140.386593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627140.386595:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627140.386597:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627140.577755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627140.577961:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627141.345788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627141.601812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627141.602101:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627142.369801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627142.625773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627142.625846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627143.393797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627143.649777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627143.649865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627144.417736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627144.673782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627144.673851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627145.441740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627145.505784:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627145.505796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627145.505801:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627145.505810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11480 00000100:00080000:1.0:1778627145.505822:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627145.505834:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627145.505866:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828478464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627145.505886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627145.505889:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627145.505890:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627145.505893:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11480 00000100:00080000:1.0:1778627145.505896:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627145.505898:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627145.505906:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627145.505917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828478592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627145.505922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828478464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627145.505928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627145.505957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627145.505958:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627145.505962:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11480 00000100:00080000:1.0:1778627145.505967:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627145.505969:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627145.505974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627145.505984:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828478720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627145.505988:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627145.505996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627145.505999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627145.506000:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627145.506004:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11480 00000100:00080000:1.0:1778627145.506007:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627145.506009:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627145.506012:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627145.506022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828478848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627145.506022:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627145/real 1778627145] req@ffff954350197100 x1865013828478464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627145.506055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627145.506059:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778627145.506062:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828478464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627145.506063:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627145.506073:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11480 00000100:00080000:2.0:1778627145.506077:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627145.506078:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627145.506084:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627145.506086:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627145.506089:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828478464:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627145.506099:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627145.506109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627145.506114:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828478976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627145.506141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627145.506161:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627145/real 1778627145] req@ffff954350194000 x1865013828478592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627145.506170:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828478592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627145.506176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627145.506179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627145.506181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627145.506185:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627145.506196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627145.506202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627145/real 1778627145] req@ffff954350196d80 x1865013828478720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627145.506209:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828478720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627145.506214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627145.506216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627145.506218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627145.506222:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627145.506231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627145.506237:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627145/real 1778627145] req@ffff954350197480 x1865013828478848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627145.506254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828478848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627145.506259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627145.506261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627145.506263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627145.506275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627145.506286:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627145.506292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627145/real 1778627145] req@ffff954350194700 x1865013828478976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627145.506299:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828478976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627145.506304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627145.506306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627145.506308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828478976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627145.697747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627145.697810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627146.465784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627146.721750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627146.721810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627147.489743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627147.745769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627147.745828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627148.513770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627148.769758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627148.769880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627149.537754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627149.793803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627149.793875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627150.561768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627150.625722:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627150.625730:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627150.625734:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627150.625741:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11485 00000100:00080000:1.0:1778627150.625750:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627150.625755:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627150.625788:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828479104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627150.625815:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627150.625820:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627150.625822:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627150.625826:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11485 00000100:00080000:1.0:1778627150.625840:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627150.625845:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627150.625855:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627150.625869:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828479232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627150.625870:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828479104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627150.625884:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627150.625886:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627150.625888:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627150.625890:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11485 00000100:00080000:1.0:1778627150.625904:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627150.625908:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627150.625912:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627150.625920:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828479360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627150.625928:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627150.625930:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627150.625932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627150.625934:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11485 00000100:00080000:1.0:1778627150.625936:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627150.625938:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627150.625941:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627150.625945:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627150.625952:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828479488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627150.625961:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627150.625964:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627150.625966:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627150.625970:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11485 00000100:00080000:1.0:1778627150.625974:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627150.625978:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627150.625984:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627150.625988:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627150/real 1778627150] req@ffff954350197800 x1865013828479104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627150.625999:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828479616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627150.626020:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828479104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627150.626047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627150.626053:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627150.626059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828479104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627150.626084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627150.626108:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627150.626122:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627150/real 1778627150] req@ffff954350194380 x1865013828479232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627150.626136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828479232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627150.626146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627150.626150:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627150.626163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627150.626170:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627150.626192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627150.626203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627150/real 1778627150] req@ffff954350195500 x1865013828479360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627150.626217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828479360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627150.626240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627150.626244:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627150.626248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627150.626255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627150.626278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627150.626287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627150/real 1778627150] req@ffff954350195c00 x1865013828479488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627150.626295:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828479488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627150.626300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627150.626303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627150.626305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627150.626309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627150.626320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627150.626325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627150/real 1778627150] req@ffff954350194a80 x1865013828479616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627150.626332:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828479616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627150.626338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627150.626339:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627150.626346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627150.817733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627150.817819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627151.585748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627151.841819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627151.841915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627152.609729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627152.865800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627152.865875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627153.633733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627153.889803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627153.889886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627154.657850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627154.913837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627154.913921:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627155.681798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627155.745773:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627155.745781:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627155.745785:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627155.745793:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11490 00000100:00080000:1.0:1778627155.745803:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627155.745817:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627155.745853:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828479744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627155.745881:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627155.745884:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627155.745886:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627155.745889:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11490 00000100:00080000:1.0:1778627155.745892:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627155.745895:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627155.745903:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627155.745914:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828479872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627155.745922:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627155.745924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627155.745926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627155.745930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11490 00000100:00100000:2.0:1778627155.745932:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828479744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627155.745933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627155.745938:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627155.745956:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627155.745965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828480000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627155.745973:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627155.745975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627155.745977:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627155.745979:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11490 00000100:00080000:1.0:1778627155.745983:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627155.745985:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627155.745989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627155.745996:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828480128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627155.746001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627155.746003:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627155.746006:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778627155.746008:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627155.746010:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11490 00000100:00080000:1.0:1778627155.746014:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627155.746017:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627155.746021:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627155.746028:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828480256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627155.746049:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627155/real 1778627155] req@ffff954350194e00 x1865013828479744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627155.746070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828479744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627155.746085:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627155.746088:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627155.746092:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828479744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627155.746111:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627155.746128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627155.746136:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627155/real 1778627155] req@ffff954350195880 x1865013828479872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627155.746144:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828479872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627155.746151:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627155.746155:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627155.746166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828479872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627155.746173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627155.746196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627155.746209:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627155/real 1778627155] req@ffff954350197b80 x1865013828480000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627155.746222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828480000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627155.746230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627155.746234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627155.746237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627155.746244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627155.746275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627155.746289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627155/real 1778627155] req@ffff954350195180 x1865013828480128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627155.746303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828480128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627155.746311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627155.746315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627155.746318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627155.746324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627155.746347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627155.746368:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627155/real 1778627155] req@ffff954350196a00 x1865013828480256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627155.746382:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828480256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627155.746391:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627155.746394:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627155.746398:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627155.937759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627155.937812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627156.705790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627156.961784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627156.961830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627157.729749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627157.985778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627157.985809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627158.753754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627159.009768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627159.009809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627159.777780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627160.033747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627160.033790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627160.801774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627160.865767:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627160.865775:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627160.865794:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627160.865802:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11495 00000100:00080000:1.0:1778627160.865811:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627160.865817:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627160.865845:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828480384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627160.865870:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627160.865872:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627160.865874:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627160.865877:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11495 00000100:00080000:1.0:1778627160.865879:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627160.865882:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627160.865889:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627160.865899:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828480512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627160.865907:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627160.865910:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627160.865911:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627160.865913:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11495 00000100:00080000:1.0:1778627160.865916:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627160.865918:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828480384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627160.865921:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627160.865928:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627160.865937:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828480640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627160.865961:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627160.865965:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627160.865967:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627160.865974:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11495 00000800:00000100:2.0:1778627160.865975:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627160.865982:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627160.865988:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627160.865995:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627160.866005:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627160/real 1778627160] req@ffff954350195f80 x1865013828480384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627160.866009:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828480768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627160.866032:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627160.866035:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627160.866036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627160.866041:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11495 00000100:00100000:2.0:1778627160.866042:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828480384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627160.866044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627160.866050:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627160.866054:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627160.866064:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828480896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627160.866064:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627160.866075:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627160.866082:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828480384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627160.866092:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627160.866119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627160.866141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627160/real 1778627160] req@ffff954350196a00 x1865013828480512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627160.866154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828480512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627160.866163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627160.866166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627160.866170:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627160.866194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627160.866213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627160.866224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627160/real 1778627160] req@ffff954350195180 x1865013828480640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627160.866237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828480640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627160.866246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627160.866249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627160.866253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627160.866260:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627160.866281:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627160.866292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627160/real 1778627160] req@ffff954350197b80 x1865013828480768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627160.866305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828480768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627160.866320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627160.866323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627160.866327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627160.866333:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627160.866355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627160.866366:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627160/real 1778627160] req@ffff954350195880 x1865013828480896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627160.866379:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828480896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627160.866388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627160.866392:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627160.866396:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828480896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627161.057785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627161.057844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627161.825732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627162.081770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627162.081822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627162.849753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627163.105742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627163.105792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627163.873744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627164.129761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627164.129823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627164.897737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627165.153744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627165.153791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627165.921782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627165.985756:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627165.985765:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627165.985768:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627165.985776:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11500 00000100:00080000:3.0:1778627165.985785:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627165.985791:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627165.985837:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828481024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627165.985859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627165.985862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627165.985863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627165.985867:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11500 00000100:00080000:3.0:1778627165.985869:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627165.985872:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627165.985879:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627165.985888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828481152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627165.985895:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627165.985897:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627165.985899:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627165.985901:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11500 00000100:00080000:3.0:1778627165.985903:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627165.985907:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627165.985908:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828481024:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778627165.985910:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627165.985920:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828481280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627165.985927:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627165.985929:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627165.985931:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627165.985938:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11500 00000100:00080000:3.0:1778627165.985941:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627165.985944:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627165.985948:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627165.985957:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828481408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627165.985964:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627165.985966:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627165.985967:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627165.985969:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11500 00000100:00080000:3.0:1778627165.985971:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627165.985976:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627165.985977:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627165.985982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627165.985988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828481536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627165.986011:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627165/real 1778627165] req@ffff95434701d500 x1865013828481024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627165.986027:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828481024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627165.986041:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627165.986044:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627165.986047:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828481024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627165.986053:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627165.986069:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627165.986077:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627165/real 1778627165] req@ffff95434701c380 x1865013828481152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627165.986085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828481152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627165.986091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627165.986093:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627165.986095:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627165.986112:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627165.986122:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627165.986128:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627165/real 1778627165] req@ffff95434701f100 x1865013828481280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627165.986135:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828481280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627165.986140:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627165.986142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627165.986144:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627165.986153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627165.986179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627165.986191:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627165/real 1778627165] req@ffff95434701f800 x1865013828481408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627165.986203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828481408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627165.986213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627165.986216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627165.986219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627165.986226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627165.986243:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627165.986250:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627165/real 1778627165] req@ffff95434701c700 x1865013828481536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627165.986257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828481536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627165.986263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627165.986265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627165.986267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627166.177780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627166.177840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627166.945740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627167.201751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627167.201800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627167.969781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627168.225771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627168.225830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627168.993762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627169.249837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627169.249879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627170.017820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627170.273803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627170.273899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627171.041746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627171.105801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627171.105812:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627171.105816:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627171.105827:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11505 00000100:00080000:1.0:1778627171.105872:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627171.105881:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627171.105929:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828481664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627171.105972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627171.105976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627171.105977:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627171.105982:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11505 00000100:00080000:1.0:1778627171.105985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627171.105989:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627171.105999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627171.106011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828481792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627171.106020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627171.106024:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627171.106026:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627171.106026:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828481664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627171.106039:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11505 00000100:00080000:1.0:1778627171.106042:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627171.106045:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627171.106050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627171.106064:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828481920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627171.106074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627171.106085:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627171.106087:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627171.106092:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11505 00000100:00080000:1.0:1778627171.106097:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627171.106102:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627171.106109:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627171.106124:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828482048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627171.106133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627171.106136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627171.106138:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627171.106142:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481792:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627171.106145:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11505 00000100:00080000:1.0:1778627171.106151:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627171.106155:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627171.106165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627171.106176:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828482176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627171.106182:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627171.106211:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627171.106235:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627171.297794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627171.297877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627172.065826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627172.321784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627172.321873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627173.089822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627173.345774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627173.345853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627174.113837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627174.177995:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627174.178041:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627174.178062:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627174.178136:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627174.178148:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627174.178158:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627174.178169:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778627174.178206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627171/real 1778627174] req@ffff954350194380 x1865013828482176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627174.178242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828482176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627174.178257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627174.178265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627174.178270:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627174.178279:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627171/real 1778627174] req@ffff954350194a80 x1865013828481792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627174.178292:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828481792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627174.178301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627174.178317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627174.178321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627174.178329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627171/real 1778627174] req@ffff954350195c00 x1865013828481920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627174.178345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828481920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627174.178353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627174.178357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627174.178361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828481920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627174.178385:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627171/real 1778627174] req@ffff954350195500 x1865013828482048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627174.178400:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828482048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627174.178409:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627174.178413:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627174.178416:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627174.178423:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627171/real 1778627174] req@ffff954350194e00 x1865013828481664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627174.178435:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828481664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627174.178443:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627174.178453:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627174.178457:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828481664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627174.369819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627174.369891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627175.137789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627175.393779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627175.393844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627176.161758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627176.225772:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627176.225783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627176.225787:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627176.225797:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11510 00000100:00080000:1.0:1778627176.225806:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627176.225812:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627176.225841:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828482304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627176.225873:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627176.225876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627176.225878:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627176.225881:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11510 00000100:00080000:1.0:1778627176.225885:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627176.225889:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627176.225899:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627176.225916:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828482432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627176.225922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828482304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627176.225937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627176.225940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627176.225942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627176.225947:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11510 00000100:00080000:1.0:1778627176.225951:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627176.225955:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627176.225961:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627176.225974:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828482560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627176.225984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627176.225987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627176.225989:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778627176.225995:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627176.225999:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11510 00000100:00080000:1.0:1778627176.226013:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627176.226019:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627176.226026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627176.226040:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627176/real 1778627176] req@ffff954350197800 x1865013828482304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627176.226042:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828482688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627176.226078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627176.226081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627176.226084:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627176.226087:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828482304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627176.226091:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11510 00000100:00080000:1.0:1778627176.226107:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627176.226111:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627176.226119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778627176.226120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627176.226128:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627176.226134:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828482304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627176.226138:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828482816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627176.226177:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627176.226206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627176.226219:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627176/real 1778627176] req@ffff954350194700 x1865013828482432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627176.226233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828482432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627176.226243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627176.226246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627176.226250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627176.226266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627176.226291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627176.226304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627176/real 1778627176] req@ffff954350197480 x1865013828482560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627176.226315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828482560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627176.226323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627176.226340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627176.226344:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627176.226352:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627176.226374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627176.226386:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627176/real 1778627176] req@ffff954350196d80 x1865013828482688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627176.226399:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828482688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627176.226408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627176.226411:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627176.226414:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627176.226420:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627176.226440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627176.226451:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627176/real 1778627176] req@ffff954350194000 x1865013828482816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627176.226475:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828482816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627176.226485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627176.226488:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627176.226492:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828482816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627176.417728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627176.417803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627177.185787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627177.441743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627177.441824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627178.209833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627178.465863:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627178.465914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627179.233819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627179.489762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627179.489837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627180.257814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627180.513802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627180.513889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627181.281794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627181.345793:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627181.345800:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627181.345804:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627181.345812:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11515 00000100:00080000:0.0:1778627181.345822:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627181.345828:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627181.345858:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828482944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627181.345882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627181.345885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627181.345886:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627181.345889:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11515 00000100:00080000:0.0:1778627181.345892:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627181.345894:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627181.345901:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627181.345911:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828483072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627181.345918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627181.345921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627181.345922:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627181.345925:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11515 00000100:00080000:0.0:1778627181.345926:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627181.345946:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627181.345946:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828482944:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627181.345952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627181.345963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828483200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627181.345972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627181.345976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627181.345979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627181.345983:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11515 00000100:00080000:0.0:1778627181.345988:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627181.345993:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627181.345999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627181.346012:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828483328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627181.346023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627181.346027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627181.346029:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627181.346032:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11515 00000100:00080000:0.0:1778627181.346037:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627181.346040:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627181.346044:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627181.346051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627181.346054:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828483456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627181.346110:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627181/real 1778627181] req@ffff954342f49500 x1865013828482944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627181.346135:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828482944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627181.346146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627181.346149:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627181.346153:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828482944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627181.346160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627181.346181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627181.346190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627181/real 1778627181] req@ffff954342f4ad80 x1865013828483072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627181.346212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828483072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627181.346219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627181.346221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627181.346238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627181.346266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627181.346282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627181.346290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627181/real 1778627181] req@ffff954342f48a80 x1865013828483200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627181.346298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828483200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627181.346303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627181.346306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627181.346308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627181.346312:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627181.346322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627181.346328:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627181/real 1778627181] req@ffff954342f4b100 x1865013828483328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627181.346335:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828483328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627181.346349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627181.346352:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627181.346354:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627181.346358:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627181.346372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627181.346378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627181/real 1778627181] req@ffff954342f49f80 x1865013828483456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627181.346396:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828483456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627181.346402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627181.346405:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627181.346407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627181.537813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627181.537881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627182.305814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627182.561814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627182.561916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627183.329795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627183.585789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627183.585841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627184.353745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627184.609797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627184.609821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627185.377806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627185.633793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627185.633870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627186.401754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627186.465758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627186.465768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627186.465773:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627186.465784:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11520 00000100:00080000:0.0:1778627186.465797:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627186.465803:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627186.465846:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828483584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627186.465873:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627186.465876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627186.465878:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627186.465881:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11520 00000100:00080000:0.0:1778627186.465884:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627186.465886:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627186.465894:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627186.465907:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828483712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627186.465918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627186.465922:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627186.465924:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627186.465930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11520 00000100:00100000:2.0:1778627186.465931:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828483584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627186.465955:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627186.465960:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627186.465967:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627186.465980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828483840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627186.465988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627186.465991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627186.465993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627186.465996:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11520 00000100:00080000:0.0:1778627186.465999:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627186.466001:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627186.466005:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627186.466008:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627186.466017:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828483968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627186.466025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627186.466028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627186.466030:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627186.466032:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11520 00000100:00080000:0.0:1778627186.466035:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627186.466037:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627186.466041:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627186.466048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828484096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627186.466063:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627186/real 1778627186] req@ffff954342f48700 x1865013828483584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627186.466085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828483584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627186.466108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627186.466112:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627186.466115:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828483584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627186.466133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627186.466154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627186.466164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627186/real 1778627186] req@ffff954342f4a680 x1865013828483712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627186.466173:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828483712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627186.466179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627186.466181:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627186.466184:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627186.466207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627186.466218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627186.466224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627186/real 1778627186] req@ffff954342f49f80 x1865013828483840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627186.466232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828483840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627186.466237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627186.466240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627186.466242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627186.466251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627186.466265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627186.466271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627186/real 1778627186] req@ffff954342f4b100 x1865013828483968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627186.466279:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828483968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627186.466284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627186.466286:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627186.466288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828483968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627186.466292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627186.466302:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627186.466307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627186/real 1778627186] req@ffff954342f48a80 x1865013828484096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627186.466314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828484096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627186.466320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627186.466321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627186.466323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627186.657813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627186.657896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627187.425917:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627187.681797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627187.681876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627188.449830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627188.705767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627188.705822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627189.473844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627189.729754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627189.729917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627190.497757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627190.753788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627190.753835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627191.521824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627191.585786:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627191.585796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627191.585801:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627191.585812:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11526 00000100:00080000:0.0:1778627191.585826:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627191.585832:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627191.585882:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828484224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627191.585921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627191.585926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627191.585928:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627191.585934:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11526 00000100:00080000:0.0:1778627191.585938:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627191.585943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627191.585955:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627191.585972:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828484352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627191.585981:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828484224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627191.585987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627191.585991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627191.586017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627191.586033:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11526 00000100:00080000:0.0:1778627191.586038:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627191.586041:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627191.586046:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627191.586057:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828484480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627191.586065:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627191.586068:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627191.586070:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627191.586073:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11526 00000100:00080000:0.0:1778627191.586076:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627191.586078:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627191.586083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627191.586086:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627191.586107:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828484608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627191.586120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627191.586124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627191.586126:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627191.586132:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11526 00000100:00080000:0.0:1778627191.586138:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627191.586139:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627191/real 1778627191] req@ffff954342f4ad80 x1865013828484224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627191.586144:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627191.586150:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627191.586168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828484224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627191.586176:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828484736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627191.586180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627191.586189:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627191.586195:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828484224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627191.586221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627191.586244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627191.586257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627191/real 1778627191] req@ffff954342f49500 x1865013828484352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627191.586271:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828484352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627191.586281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627191.586285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627191.586289:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627191.586295:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627191.586314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627191.586322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627191/real 1778627191] req@ffff954342f4bb80 x1865013828484480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627191.586330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828484480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627191.586355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627191.586357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627191.586360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627191.586364:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627191.586385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627191.586397:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627191/real 1778627191] req@ffff954342f4b480 x1865013828484608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627191.586410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828484608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627191.586419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627191.586422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627191.586425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627191.586431:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627191.586452:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627191.586464:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627191/real 1778627191] req@ffff954342f4a300 x1865013828484736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627191.586477:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828484736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627191.586484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627191.586487:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627191.586495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627191.777797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627191.777847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627192.545739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627192.801742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627192.801820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627193.569793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627193.825758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627193.825810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627194.593749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627194.849738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627194.849784:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627195.617761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627195.873761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627195.873831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627196.641783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627196.705777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627196.705806:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627196.705810:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627196.705821:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11531 00000100:00080000:1.0:1778627196.705834:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627196.705840:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627196.705894:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828484864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627196.705947:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627196.705953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627196.705955:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627196.705960:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11531 00000100:00080000:1.0:1778627196.705965:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627196.705971:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627196.705984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627196.705996:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828484864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627196.706006:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828484992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627196.706020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627196.706023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627196.706025:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627196.706030:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11531 00000100:00080000:1.0:1778627196.706034:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627196.706038:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627196.706042:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627196.706051:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828485120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627196.706064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627196.706067:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627196.706068:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627196.706071:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11531 00000100:00080000:1.0:1778627196.706075:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627196.706078:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627196.706083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627196.706083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627196.706094:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828485248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627196.706103:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627196.706106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627196.706107:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627196.706109:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11531 00000100:00080000:1.0:1778627196.706112:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627196.706115:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627196.706118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627196.706125:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627196/real 1778627196] req@ffff954350197100 x1865013828484864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627196.706132:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828485376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627196.706150:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828484864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627196.706161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627196.706166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627196.706169:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828484864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627196.706192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627196.706208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627196.706216:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627196/real 1778627196] req@ffff954350194000 x1865013828484992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627196.706225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828484992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627196.706231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627196.706233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627196.706235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828484992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627196.706239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627196.706249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627196.706262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627196/real 1778627196] req@ffff954350196d80 x1865013828485120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627196.706272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828485120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627196.706282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627196.706285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627196.706288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627196.706294:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627196.706317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627196.706328:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627196/real 1778627196] req@ffff954350197480 x1865013828485248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627196.706339:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828485248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627196.706345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627196.706347:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627196.706349:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627196.706354:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627196.706367:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627196.706373:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627196/real 1778627196] req@ffff954350194700 x1865013828485376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627196.706381:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828485376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627196.706392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627196.706394:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627196.706396:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627196.897734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627196.897814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627197.665839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627197.921840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627197.921918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627198.689792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627198.945738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627198.945841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627199.713810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000008:00020000:1.0:1778627199.777775:0:157327:0:(osc_cache.c:1001:osc_extent_wait()) extent ffff9543460e2300@{[0 -> 0/1023], [3|0|+|rpc|wihY|ffff954352f20c40], [28672|1|+|-|ffff95434372fa00|1024|ffff9543580fa7c0]} lustre-OST0000-osc-ffff9543436d7000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1778627199.777801:0:157326:0:(osc_cache.c:1001:osc_extent_wait()) extent ffff9543460e2900@{[0 -> 0/1023], [3|0|+|rpc|wihY|ffff954352f20fc0], [28672|1|+|-|ffff95434372f200|1024|ffff9543429acf80]} lustre-OST0001-osc-ffff9543436d7000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:1.0:1778627199.777825:0:157327:0:(osc_cache.c:1001:osc_extent_wait()) ### extent: 000000009e8cc664 ns: lustre-OST0000-osc-ffff9543436d7000 lock: 00000000273a5d3a/0x56379fd28fe4ee83 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x19d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604834 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00020000:2.0:1778627199.795972:0:157326:0:(osc_cache.c:1001:osc_extent_wait()) ### extent: 00000000a9814215 ns: lustre-OST0001-osc-ffff9543436d7000 lock: 000000002decd78f/0x56379fd28fe4eea6 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x1994:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xa0dae0fc83604896 expref: -99 pid: 188183 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000100:00100000:3.0:1778627199.969803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627199.969925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627200.737765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627200.993726:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627200.993830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627201.761756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627201.825810:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627201.825832:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627201.825836:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627201.825848:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11536 00000100:00080000:0.0:1778627201.825861:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627201.825869:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627201.825908:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828485504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627201.825940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627201.825945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627201.825947:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627201.825952:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11536 00000100:00080000:0.0:1778627201.825956:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627201.825960:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627201.825982:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828485504:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627201.825987:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627201.826006:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828485632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627201.826019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627201.826023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627201.826026:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627201.826029:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11536 00000100:00080000:0.0:1778627201.826035:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627201.826035:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627201.826043:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627201.826050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627201.826063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828485760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627201.826069:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627201/real 1778627201] req@ffff954342f49880 x1865013828485504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627201.826085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627201.826090:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627201.826092:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627201.826094:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828485504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627201.826103:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11536 00000100:00080000:2.0:1778627201.826106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627201.826109:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627201.826112:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627201.826117:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828485504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627201.826118:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627201.826127:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627201.826138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627201.826141:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828485888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627201.826155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778627201.826159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627201.826163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778627201.826170:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627201/real 1778627201] req@ffff954342f49c00 x1865013828485632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627201.826173:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627201.826183:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11536 00000100:00080000:0.0:1778627201.826187:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627201.826195:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627201.826204:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627201.826206:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828485632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627201.826214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627201.826217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627201.826222:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828486016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627201.826222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627201.826233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627201.826254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627201.826267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627201/real 1778627201] req@ffff954342f49180 x1865013828485760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627201.826280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828485760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627201.826288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627201.826292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627201.826295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627201.826312:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627201.826334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627201.826346:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627201/real 1778627201] req@ffff954342f48000 x1865013828485888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627201.826359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828485888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627201.826381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627201.826383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627201.826386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828485888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627201.826390:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627201.826405:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627201.826412:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627201/real 1778627201] req@ffff954342f48380 x1865013828486016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627201.826421:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828486016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627201.826431:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627201.826434:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627201.826438:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627202.017761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627202.017852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627202.785781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627203.041900:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627203.042873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627203.809803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627204.065823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627204.065978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627204.833775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627205.090000:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778627205.090043:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627205.857798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627206.113914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778627206.113914:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627206.881813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627206.945799:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627206.945810:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627206.945816:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627206.945826:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11541 00000100:00080000:0.0:1778627206.945838:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627206.945845:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627206.945888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828486144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627206.945916:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627206.945920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627206.945921:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627206.945924:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11541 00000100:00080000:0.0:1778627206.945945:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627206.945949:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627206.945960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627206.945970:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828486272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627206.945982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627206.945984:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828486144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627206.945986:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627206.945991:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627206.945999:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11541 00000100:00080000:0.0:1778627206.946004:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627206.946011:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627206.946017:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627206.946037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828486400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627206.946049:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627206.946052:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627206.946054:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627206.946057:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11541 00000100:00080000:0.0:1778627206.946061:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627206.946064:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627206.946069:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627206.946076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627206.946092:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828486528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627206.946114:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627206/real 1778627206] req@ffff954342f4aa00 x1865013828486144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627206.946121:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627206.946135:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627206.946139:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627206.946147:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11541 00000100:00080000:0.0:1778627206.946153:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627206.946158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828486144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627206.946165:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627206.946180:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627206.946184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627206.946190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627206.946195:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828486656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627206.946196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828486144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627206.946237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627206.946258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627206.946267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627206/real 1778627206] req@ffff954342f48e00 x1865013828486272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627206.946276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828486272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627206.946283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627206.946285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627206.946288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627206.946292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627206.946308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627206.946318:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627206/real 1778627206] req@ffff954342f48380 x1865013828486400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627206.946331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828486400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627206.946340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627206.946345:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627206.946348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627206.946355:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627206.946373:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627206.946380:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627206/real 1778627206] req@ffff954342f48000 x1865013828486528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627206.946394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828486528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627206.946400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627206.946403:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627206.946405:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627206.946409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627206.946422:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627206.946428:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627206/real 1778627206] req@ffff954342f49180 x1865013828486656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627206.946435:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828486656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627206.946441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627206.946443:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627206.946445:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627207.137864:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627207.137964:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627207.905818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627208.161852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627208.162022:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627208.929788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627209.185821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627209.185871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627209.953742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627210.209758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627210.209813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627210.977770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627211.233767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627211.233829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627212.001733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627212.065782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627212.065790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627212.065794:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627212.065802:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11546 00000100:00080000:1.0:1778627212.065817:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627212.065824:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627212.065859:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828486784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627212.065880:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627212.065882:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627212.065884:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627212.065902:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11546 00000100:00080000:1.0:1778627212.065905:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627212.065909:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627212.065917:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627212.065931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828486912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627212.065943:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828486784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627212.065952:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627212.065955:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627212.065956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627212.065960:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11546 00000100:00080000:1.0:1778627212.065962:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627212.065965:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627212.065969:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627212.065977:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828487040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627212.065984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627212.065986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627212.065987:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627212.065995:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11546 00000100:00080000:1.0:1778627212.066000:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627212.066003:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627212.066006:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627212.066013:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828487168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627212.066017:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627212.066031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627212.066034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627212.066036:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627212.066040:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11546 00000100:00080000:1.0:1778627212.066043:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627212.066046:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627212.066050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627212.066061:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627212/real 1778627212] req@ffff954350197800 x1865013828486784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627212.066064:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828487296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627212.066102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828486784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627212.066128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627212.066133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627212.066138:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828486784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627212.066162:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627212.066189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627212.066202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627212/real 1778627212] req@ffff954350194e00 x1865013828486912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627212.066216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828486912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627212.066225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627212.066229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627212.066232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828486912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627212.066239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627212.066262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627212.066273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627212/real 1778627212] req@ffff954350195500 x1865013828487040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627212.066286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828487040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627212.066295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627212.066299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627212.066302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627212.066319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627212.066343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627212.066354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627212/real 1778627212] req@ffff954350195c00 x1865013828487168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627212.066368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828487168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627212.066383:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627212.066388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627212.066391:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627212.066398:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627212.066419:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627212.066430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627212/real 1778627212] req@ffff954350194a80 x1865013828487296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627212.066443:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828487296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627212.066452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627212.066456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627212.066459:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627212.257755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627212.257841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627213.025777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627213.281757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627213.281828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627214.049771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627214.305741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627214.305850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627215.073816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627215.329766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627215.329869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627216.097767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627216.353758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627216.353853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627217.121828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627217.185826:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627217.185837:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627217.185843:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627217.185876:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11551 00000100:00080000:2.0:1778627217.185888:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627217.185894:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627217.185937:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828487424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627217.185969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627217.185974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627217.185975:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627217.185979:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11551 00000100:00080000:2.0:1778627217.185982:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627217.185985:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627217.185994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627217.186005:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828487552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627217.186016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627217.186020:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627217.186022:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627217.186027:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11551 00000100:00080000:2.0:1778627217.186032:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627217.186034:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828487424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627217.186040:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627217.186048:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627217.186075:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828487680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627217.186095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627217.186098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627217.186100:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627217.186104:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11551 00000100:00080000:2.0:1778627217.186108:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627217.186111:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627217.186116:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627217.186121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627217.186130:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828487808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627217.186137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627217.186140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627217.186142:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627217.186145:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11551 00000100:00080000:2.0:1778627217.186148:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627217.186150:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627217.186153:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627217.186161:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828487936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627217.186173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627217/real 1778627217] req@ffff954358789880 x1865013828487424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627217.186214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828487424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627217.186227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627217.186233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627217.186238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828487424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627217.186246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627217.186271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627217.186284:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627217/real 1778627217] req@ffff954358788700 x1865013828487552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627217.186298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828487552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627217.186307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627217.186310:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627217.186314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627217.186346:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627217.186366:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627217.186378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627217/real 1778627217] req@ffff95435878b100 x1865013828487680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627217.186391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828487680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627217.186413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627217.186418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627217.186421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627217.186428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627217.186449:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627217.186476:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627217/real 1778627217] req@ffff95435878bb80 x1865013828487808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627217.186490:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828487808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627217.186499:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627217.186503:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627217.186507:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627217.186514:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627217.186549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627217.186560:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627217/real 1778627217] req@ffff95435878b480 x1865013828487936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627217.186572:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828487936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627217.186582:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627217.186586:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627217.186596:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828487936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627217.377774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627217.377849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627218.145836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627218.401758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627218.401820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627219.169799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627219.425779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627219.425869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627220.193823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627220.449795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627220.449855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627221.217808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627221.473840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627221.474068:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627222.241841:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627222.305745:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627222.305755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627222.305760:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627222.305770:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11556 00000100:00080000:2.0:1778627222.305804:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627222.305810:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627222.305849:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828488064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627222.305877:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627222.305880:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627222.305882:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627222.305885:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11556 00000100:00080000:2.0:1778627222.305891:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627222.305894:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627222.305903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627222.305912:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828488192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627222.305919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627222.305922:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627222.305924:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627222.305926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11556 00000100:00080000:2.0:1778627222.305928:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627222.305930:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627222.305933:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627222.305943:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828488320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627222.305945:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828488064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627222.305954:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627222.305961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627222.305964:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627222.305969:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11556 00000100:00080000:2.0:1778627222.305973:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627222.305978:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627222.305985:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627222.306000:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828488448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627222.306012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627222.306015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627222.306017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627222.306020:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11556 00000100:00080000:2.0:1778627222.306023:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627222.306026:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627222.306030:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627222.306032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627222.306041:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828488576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627222.306079:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627222/real 1778627222] req@ffff95435878aa00 x1865013828488064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627222.306100:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828488064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627222.306112:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627222.306117:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627222.306122:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828488064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627222.306131:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627222.306154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627222.306167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627222/real 1778627222] req@ffff954358788380 x1865013828488192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627222.306180:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828488192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627222.306190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627222.306194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627222.306206:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627222.306239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627222.306260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627222.306281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627222/real 1778627222] req@ffff95435878b480 x1865013828488320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627222.306296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828488320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627222.306305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627222.306309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627222.306313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627222.306320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627222.306341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627222.306351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627222/real 1778627222] req@ffff95435878bb80 x1865013828488448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627222.306364:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828488448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627222.306373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627222.306377:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627222.306381:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627222.306388:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627222.306406:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627222.306444:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627222/real 1778627222] req@ffff95435878b100 x1865013828488576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627222.306457:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828488576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627222.306468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627222.306472:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627222.306476:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627222.497809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627222.497885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627223.265776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627223.521787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627223.521832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627224.289781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627224.545813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627224.545888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627225.313832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627225.569800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627225.569866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627226.337788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627226.593805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627226.593857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627227.361764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627227.425770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627227.425781:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627227.425786:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627227.425796:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11561 00000100:00080000:2.0:1778627227.425807:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627227.425815:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627227.425851:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828488704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627227.425899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627227.425902:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627227.425904:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627227.425907:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11561 00000100:00080000:2.0:1778627227.425910:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627227.425913:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627227.425920:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627227.425930:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828488832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627227.425938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627227.425941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627227.425942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627227.425949:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11561 00000100:00080000:2.0:1778627227.425952:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627227.425956:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627227.425958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828488704:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778627227.425960:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627227.425970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828488960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627227.425978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627227.425980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627227.425981:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627227.425984:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11561 00000100:00080000:2.0:1778627227.425993:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627227.425997:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627227.426001:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627227.426010:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828489088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627227.426023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627227.426031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1778627227.426032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627227.426033:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627227.426040:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11561 00000100:00080000:2.0:1778627227.426043:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627227.426045:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627227.426049:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627227.426058:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828489216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627227.426082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627227/real 1778627227] req@ffff954358788700 x1865013828488704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627227.426102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828488704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627227.426114:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627227.426118:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627227.426123:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828488704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627227.426131:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627227.426153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627227.426173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627227/real 1778627227] req@ffff954358789880 x1865013828488832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627227.426185:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828488832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627227.426195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627227.426198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627227.426202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627227.426226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627227.426242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627227.426251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627227/real 1778627227] req@ffff954358789c00 x1865013828488960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627227.426263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828488960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627227.426272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627227.426274:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627227.426278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828488960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627227.426283:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627227.426299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627227.426305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627227/real 1778627227] req@ffff954358789500 x1865013828489088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627227.426329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828489088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627227.426345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627227.426349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627227.426352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627227.426359:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627227.426381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627227.426392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627227/real 1778627227] req@ffff954358788e00 x1865013828489216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627227.426405:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828489216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627227.426414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627227.426418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627227.426421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627227.617804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627227.617867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627228.385764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627228.641803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627228.641868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627229.409752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627229.665762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627229.665823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627230.433773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627230.689835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627230.689947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627231.457747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627231.713764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627231.713815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627232.481812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627232.545797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627232.545805:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627232.545809:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627232.545818:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11567 00000100:00080000:2.0:1778627232.545827:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627232.545833:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627232.545864:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828489344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627232.545886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627232.545904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627232.545906:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627232.545910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11567 00000100:00080000:2.0:1778627232.545914:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627232.545918:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627232.545924:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627232.545936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828489472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627232.545944:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828489344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627232.545955:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627232.545958:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627232.545959:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627232.545962:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11567 00000100:00080000:2.0:1778627232.545964:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627232.545967:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627232.545971:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627232.545979:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828489600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627232.545986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627232.545989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627232.545990:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627232.545995:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11567 00000100:00080000:2.0:1778627232.546002:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627232.546004:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627232.546013:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627232.546023:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828489728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627232.546033:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627232.546036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627232.546039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627232.546040:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627232.546043:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11567 00000100:00080000:2.0:1778627232.546047:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627232.546051:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627232.546055:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627232.546064:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828489856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627232.546082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627232/real 1778627232] req@ffff954358789180 x1865013828489344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627232.546103:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828489344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627232.546129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627232.546136:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627232.546140:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828489344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627232.546180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627232.546201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627232.546225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627232/real 1778627232] req@ffff954358788a80 x1865013828489472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627232.546236:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828489472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627232.546242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627232.546245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627232.546247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627232.546252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627232.546266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627232.546273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627232/real 1778627232] req@ffff95435878b800 x1865013828489600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627232.546281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828489600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627232.546286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627232.546287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627232.546289:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627232.546305:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627232.546322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627232.546329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627232/real 1778627232] req@ffff95435878a300 x1865013828489728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627232.546337:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828489728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627232.546343:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627232.546346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627232.546348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627232.546351:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627232.546362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627232.546367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627232/real 1778627232] req@ffff95435878ad80 x1865013828489856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627232.546374:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828489856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627232.546380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627232.546382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627232.546383:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828489856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627232.737802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627232.737889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627233.505766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627233.761807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627233.761946:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627234.529836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627234.785756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627234.785898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627235.553826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627235.809735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627235.809824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627236.577759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627236.833784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627236.833858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627237.601815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627237.665799:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627237.665807:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627237.665810:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627237.665818:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11572 00000100:00080000:0.0:1778627237.665827:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627237.665833:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627237.665865:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828489984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627237.665890:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627237.665893:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627237.665895:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627237.665898:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11572 00000100:00080000:0.0:1778627237.665900:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627237.665903:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627237.665910:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627237.665919:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828490112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627237.665925:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828489984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627237.665928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627237.665932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627237.665934:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627237.665937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11572 00000100:00080000:0.0:1778627237.665939:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627237.665941:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627237.665959:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627237.665967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828490240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627237.665974:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627237.665976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627237.665977:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627237.665980:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11572 00000100:00080000:0.0:1778627237.665983:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627237.665985:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627237.665989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627237.665998:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828490368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627237.666004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627237.666007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627237.666008:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627237.666013:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11572 00000100:00100000:2.0:1778627237.666013:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490112:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778627237.666015:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627237.666021:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627237.666025:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627237.666033:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828490496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627237.666056:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627237.666081:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627237.666104:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627237.857743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627237.857804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627238.625757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627238.881743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627238.881785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627239.649760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627239.905756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627239.905781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627240.673765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778627240.737963:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778627240.738013:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778627240.738030:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627240.738080:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627240.738090:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627240.738098:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627240.738105:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778627240.738187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627237/real 1778627240] req@ffff954342f4bb80 x1865013828490496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627240.738213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828490496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627240.738248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627240.738254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627240.738259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627240.738267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627237/real 1778627240] req@ffff954342f49880 x1865013828490112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627240.738282:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828490112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627240.738290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627240.738295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627240.738298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627240.738306:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627237/real 1778627240] req@ffff954342f4a300 x1865013828490240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627240.738317:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828490240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627240.738326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627240.738330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627240.738333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627240.738340:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627237/real 1778627240] req@ffff954342f4b480 x1865013828490368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627240.738352:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828490368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627240.738367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627240.738372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627240.738375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627240.738382:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627237/real 1778627240] req@ffff954342f49c00 x1865013828489984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627240.738395:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828489984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627240.738403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627240.738406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627240.738409:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828489984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627240.929739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627240.929799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627241.697829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627241.953763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627241.953867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627242.721736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627242.785739:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627242.785748:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627242.785751:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627242.785759:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11577 00000100:00080000:0.0:1778627242.785770:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627242.785776:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627242.785805:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828490624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627242.785833:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627242.785837:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627242.785839:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627242.785843:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11577 00000100:00080000:0.0:1778627242.785847:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627242.785850:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627242.785860:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627242.785877:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828490752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627242.785884:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828490624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627242.785894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627242.785897:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627242.785899:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627242.785903:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11577 00000100:00080000:0.0:1778627242.785907:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627242.785928:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627242.785934:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627242.785947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828490880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627242.785952:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627242.785963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627242.785973:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627242.785975:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627242.785981:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11577 00000100:00080000:0.0:1778627242.785987:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627242.785989:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627242/real 1778627242] req@ffff954342f49500 x1865013828490624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627242.785994:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627242.786006:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627242.786033:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828490624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627242.786042:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828491008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627242.786050:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627242.786055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627242.786059:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778627242.786061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627242.786066:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627242.786066:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828490624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627242.786078:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11577 00000100:00080000:0.0:1778627242.786083:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627242.786088:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627242.786093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627242.786106:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828491136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627242.786110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627242.786156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627242.786170:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627242/real 1778627242] req@ffff954342f4ad80 x1865013828490752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627242.786183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828490752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627242.786190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627242.786193:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627242.786196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627242.786203:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627242.786222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627242.786230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627242/real 1778627242] req@ffff954342f48a80 x1865013828490880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627242.786241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828490880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627242.786248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627242.786251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627242.786254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828490880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627242.786259:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627242.786279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627242.786289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627242/real 1778627242] req@ffff954342f4b100 x1865013828491008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627242.786314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828491008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627242.786323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627242.786326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627242.786328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627242.786348:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627242.786369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627242.786378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627242/real 1778627242] req@ffff954342f49f80 x1865013828491136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627242.786390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828491136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627242.786398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627242.786402:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627242.786405:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627242.977745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627242.977815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627243.745773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627244.001771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627244.001823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627244.769728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627245.025737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627245.025809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627245.793766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627246.049754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627246.049841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627246.817733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627247.073750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627247.073811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627247.841785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627247.905761:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627247.905769:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627247.905774:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627247.905785:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11582 00000100:00080000:0.0:1778627247.905796:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627247.905804:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627247.905847:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828491264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627247.905869:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627247.905872:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627247.905873:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627247.905879:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11582 00000100:00080000:0.0:1778627247.905882:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627247.905885:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627247.905893:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627247.905903:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828491392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627247.905922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627247.905926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627247.905928:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627247.905932:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11582 00000100:00100000:2.0:1778627247.905937:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828491264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627247.905948:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627247.905952:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627247.905957:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627247.905968:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828491520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627247.905976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627247.905978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627247.905980:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627247.905983:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11582 00000100:00080000:0.0:1778627247.905994:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627247.905999:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627247.906001:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778627247.906004:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627247.906015:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828491648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627247.906023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627247.906026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627247.906028:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627247.906033:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11582 00000100:00080000:0.0:1778627247.906038:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627247.906042:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627247/real 1778627247] req@ffff954342f4a680 x1865013828491264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627247.906046:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627247.906055:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627247.906070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828491264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627247.906072:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828491776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627247.906127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627247.906133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627247.906137:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828491264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627247.906144:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627247.906165:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627247.906175:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627247/real 1778627247] req@ffff954342f48700 x1865013828491392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627247.906183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828491392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627247.906189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627247.906191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627247.906193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627247.906212:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627247.906223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627247.906236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627247/real 1778627247] req@ffff954342f49f80 x1865013828491520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627247.906244:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828491520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627247.906249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627247.906251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627247.906253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627247.906263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627247.906278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627247.906284:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627247/real 1778627247] req@ffff954342f4b100 x1865013828491648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627247.906291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828491648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627247.906297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627247.906298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627247.906300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627247.906304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627247.906313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627247.906318:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627247/real 1778627247] req@ffff954342f48a80 x1865013828491776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627247.906325:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828491776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627247.906330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627247.906332:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627247.906333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828491776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627248.097791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627248.097847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627248.865737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627249.121737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627249.121822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627249.889738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627250.145775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627250.145832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627250.913770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627251.169734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627251.169860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627251.937826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627252.193806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627252.193914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627252.961751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627253.025751:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627253.025758:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627253.025762:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627253.025771:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11587 00000100:00080000:3.0:1778627253.025780:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627253.025793:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627253.025839:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828491904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627253.025863:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627253.025866:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627253.025867:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627253.025871:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11587 00000100:00080000:3.0:1778627253.025873:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627253.025876:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627253.025884:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627253.025911:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828491904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627253.025917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828492032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627253.025932:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627253.025936:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627253.025938:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627253.025943:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11587 00000100:00080000:3.0:1778627253.025957:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627253.025961:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627253.025965:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778627253.025966:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627253.025980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828492160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627253.025988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627253.025990:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627253.025992:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627253.025995:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11587 00000100:00080000:3.0:1778627253.025998:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627253.026001:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627253/real 1778627253] req@ffff95434701c000 x1865013828491904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627253.026003:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627253.026006:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627253.026016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828492288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627253.026024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627253.026026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627253.026028:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627253.026032:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828491904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627253.026041:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11587 00000100:00080000:3.0:1778627253.026047:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627253.026053:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627253.026057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1778627253.026057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627253.026067:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:3.0:1778627253.026072:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828492416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627253.026077:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828491904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627253.026107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627253.026141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627253.026154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627253/real 1778627253] req@ffff95434701ea00 x1865013828492032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627253.026168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828492032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627253.026178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627253.026180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627253.026182:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627253.026187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627253.026202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627253.026209:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627253/real 1778627253] req@ffff95434701d880 x1865013828492160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627253.026217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828492160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627253.026231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627253.026234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627253.026238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627253.026244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627253.026266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627253.026277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627253/real 1778627253] req@ffff95434701d180 x1865013828492288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627253.026290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828492288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627253.026299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627253.026302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627253.026305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627253.026310:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627253.026325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627253.026332:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627253/real 1778627253] req@ffff95434701df80 x1865013828492416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627253.026340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828492416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627253.026347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627253.026350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627253.026360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627253.217744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627253.217818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627253.985754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627254.241761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627254.241835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627255.009779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627255.265795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627255.265861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627256.033810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627256.289764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627256.289912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627257.057820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627257.313771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627257.313844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627258.081823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627258.145851:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627258.145862:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627258.145867:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627258.145877:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11592 00000100:00080000:0.0:1778627258.145891:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627258.145898:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627258.145943:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828492544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627258.145971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627258.145974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627258.145976:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627258.145979:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11592 00000100:00080000:0.0:1778627258.145981:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627258.145984:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627258.145993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627258.146021:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828492544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627258.146026:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828492672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627258.146038:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627258.146040:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627258.146042:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627258.146045:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11592 00000100:00080000:0.0:1778627258.146047:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627258.146050:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627258.146053:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627258.146061:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828492800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627258.146068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627258.146071:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627258.146072:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627258.146075:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11592 00000100:00080000:0.0:1778627258.146076:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627258.146079:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627258.146082:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627258.146089:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828492928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627258.146095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627258.146097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627258.146098:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627258.146105:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11592 00000100:00080000:0.0:1778627258.146108:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627258.146111:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627258.146115:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627258.146118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627258.146137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828493056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627258.146164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627258/real 1778627258] req@ffff954342f4ad80 x1865013828492544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627258.146183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828492544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627258.146193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627258.146196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627258.146199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828492544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627258.146221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627258.146238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627258.146246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627258/real 1778627258] req@ffff954342f49500 x1865013828492672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627258.146255:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828492672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627258.146261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627258.146263:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627258.146266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627258.146269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627258.146279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627258.146294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627258/real 1778627258] req@ffff954342f49c00 x1865013828492800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627258.146302:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828492800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627258.146308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627258.146310:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627258.146313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627258.146316:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627258.146334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627258.146343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627258/real 1778627258] req@ffff954342f4b480 x1865013828492928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627258.146356:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828492928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627258.146365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627258.146369:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627258.146372:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828492928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627258.146379:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627258.146401:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627258.146412:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627258/real 1778627258] req@ffff954342f4a300 x1865013828493056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627258.146421:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828493056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627258.146433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627258.146436:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627258.146438:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627258.337793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627258.337886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627259.105788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627259.361817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627259.361876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627260.129783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627260.385755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627260.385823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627261.153796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627261.409790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627261.409863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627262.177753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627262.433759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627262.433806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627263.201767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627263.265733:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627263.265741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627263.265746:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627263.265755:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11597 00000100:00080000:0.0:1778627263.265766:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627263.265772:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627263.265819:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828493184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627263.265843:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627263.265846:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627263.265847:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627263.265850:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11597 00000100:00080000:0.0:1778627263.265853:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627263.265855:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627263.265879:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627263.265891:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828493312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627263.265891:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828493184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627263.265907:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627263.265909:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627263.265910:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627263.265914:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11597 00000100:00080000:0.0:1778627263.265917:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627263.265920:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627263.265923:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627263.265931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828493440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627263.265937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627263.265940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627263.265942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627263.265944:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11597 00000100:00080000:0.0:1778627263.265947:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627263.265949:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627263.265952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627263.265955:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627263.265963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828493568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627263.265971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627263.265973:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627263.265979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627263.265982:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11597 00000100:00080000:0.0:1778627263.265985:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627263.265988:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627263.265993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627263.266008:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627263/real 1778627263] req@ffff954342f49880 x1865013828493184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627263.266013:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828493696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627263.266033:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828493184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627263.266041:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627263.266045:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627263.266048:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828493184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627263.266067:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627263.266082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627263.266090:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627263/real 1778627263] req@ffff954342f4bb80 x1865013828493312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627263.266105:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828493312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627263.266111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627263.266114:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627263.266116:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627263.266128:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627263.266142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627263.266149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627263/real 1778627263] req@ffff954342f49180 x1865013828493440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627263.266157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828493440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627263.266162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627263.266164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627263.266166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627263.266169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627263.266180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627263.266185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627263/real 1778627263] req@ffff954342f48000 x1865013828493568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627263.266192:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828493568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627263.266197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627263.266198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627263.266200:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627263.266204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627263.266230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627263.266236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627263/real 1778627263] req@ffff954342f48380 x1865013828493696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627263.266249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828493696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627263.266255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627263.266257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627263.266259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627263.457769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627263.457820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627264.225817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627264.481773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627264.481866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627265.249756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627265.505769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627265.505823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627266.273741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627266.529764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627266.529819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627267.297769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627267.553775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627267.553858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627268.321751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627268.385779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627268.385789:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627268.385794:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627268.385804:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11602 00000100:00080000:2.0:1778627268.385824:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627268.385832:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627268.385869:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828493824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627268.385903:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627268.385908:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627268.385911:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627268.385916:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11602 00000100:00080000:2.0:1778627268.385921:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627268.385927:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828493824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627268.385927:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627268.385941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627268.385957:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828493952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627268.385987:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627268.385994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627268.385998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627268.386001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627268.386006:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11602 00000100:00080000:2.0:1778627268.386011:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627268.386016:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627268/real 1778627268] req@ffff954358789f80 x1865013828493824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627268.386017:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627268.386027:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627268.386035:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828493824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627268.386043:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828494080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627268.386063:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627268.386065:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627268.386069:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778627268.386074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1778627268.386075:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828493824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627268.386077:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627268.386085:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11602 00000100:00080000:2.0:1778627268.386089:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627268.386093:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627268.386098:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493952:192.168.204.102@tcp:38:kworker.0 00000008:00100000:2.0:1778627268.386099:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627268.386117:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828494208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627268.386121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627268.386133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627268/real 1778627268] req@ffff95435878a680 x1865013828493952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627268.386134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627268.386138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627268.386142:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627268.386150:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11602 00000100:00100000:0.0:1778627268.386157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828493952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627268.386167:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627268.386168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627268.386176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778627268.386177:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627268.386181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828493952:192.168.204.102@tcp:38:kworker.0 00000008:00100000:2.0:1778627268.386186:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627268.386187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627268.386205:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828494336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627268.386211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627268.386223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627268/real 1778627268] req@ffff95435878ad80 x1865013828494080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627268.386233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828494080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627268.386239:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627268.386242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627268.386244:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627268.386255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627268.386265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627268.386271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627268/real 1778627268] req@ffff95435878a300 x1865013828494208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627268.386284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828494208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627268.386299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627268.386301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627268.386303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627268.386307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627268.386320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627268.386326:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627268/real 1778627268] req@ffff95435878b800 x1865013828494336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627268.386333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828494336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627268.386339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627268.386341:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627268.386343:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627268.577752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627268.577821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627269.345806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627269.601954:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627269.602224:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627270.369775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627270.625913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778627270.626359:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627271.393766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627271.650150:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778627271.650512:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627272.417764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627272.673794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627272.674114:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627273.441766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627273.505804:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627273.505814:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627273.505819:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627273.505829:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11608 00000100:00080000:2.0:1778627273.505840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627273.505847:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627273.505882:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828494464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627273.505910:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627273.505914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627273.505916:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627273.505920:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11608 00000100:00080000:2.0:1778627273.505924:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627273.505928:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627273.505938:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627273.505951:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828494464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627273.505952:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828494592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627273.505966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627273.505994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627273.505996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627273.506001:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11608 00000800:00000100:0.0:1778627273.506005:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627273.506009:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627273.506016:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627273.506022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627273.506038:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828494720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627273.506040:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627273/real 1778627273] req@ffff954358788a80 x1865013828494464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627273.506071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627273.506075:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828494464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627273.506076:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627273.506078:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627273.506084:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11608 00000100:00080000:2.0:1778627273.506091:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627273.506096:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627273.506096:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778627273.506102:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:2.0:1778627273.506105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627273.506106:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828494464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627273.506124:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828494848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627273.506131:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494592:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778627273.506143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627273.506147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627273.506150:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627273.506154:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11608 00000800:00000100:0.0:1778627273.506158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627273.506161:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627273.506167:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627273.506171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627273/real 1778627273] req@ffff954358789180 x1865013828494592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627273.506184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627273.506186:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828494592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627273.506195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627273.506200:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627273.506201:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828494976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627273.506204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627273.506213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627273.506233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627273.506239:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627273/real 1778627273] req@ffff954358788e00 x1865013828494720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627273.506249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828494720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627273.506255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627273.506257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627273.506259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627273.506275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627273.506288:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627273.506294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627273/real 1778627273] req@ffff954358789500 x1865013828494848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627273.506302:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828494848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627273.506310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627273.506313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627273.506316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627273.506321:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627273.506340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627273.506350:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627273/real 1778627273] req@ffff954358789c00 x1865013828494976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627273.506363:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828494976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627273.506371:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627273.506385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627273.506390:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828494976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627273.697848:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627273.698266:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627274.465744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627274.721827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627274.722111:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627275.489783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627275.745820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627275.745887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627276.513779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627276.769731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627276.769832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627277.537776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627277.793747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627277.793797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627278.561774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627278.625795:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627278.625810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627278.625816:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627278.625829:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11613 00000100:00080000:2.0:1778627278.625842:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627278.625848:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627278.625899:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828495104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627278.625927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627278.625931:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627278.625933:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627278.625938:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11613 00000100:00080000:2.0:1778627278.625942:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627278.625948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627278.625960:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627278.625976:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828495232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627278.625986:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828495104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627278.625995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627278.626006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627278.626009:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627278.626013:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11613 00000100:00080000:2.0:1778627278.626040:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627278.626046:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627278.626053:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778627278.626065:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627278.626074:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828495360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627278.626097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627278.626102:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627278/real 1778627278] req@ffff954358789880 x1865013828495104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627278.626103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627278.626107:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627278.626121:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11613 00000100:00080000:2.0:1778627278.626135:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627278.626140:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627278.626146:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627278.626147:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828495104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627278.626167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778627278.626168:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828495488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627278.626183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627278.626199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828495104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627278.626199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627278.626217:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627278.626219:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627278.626226:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11613 00000100:00080000:2.0:1778627278.626231:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627278.626234:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627278.626238:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627278.626242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627278.626254:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828495616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627278.626274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627278.626294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627278/real 1778627278] req@ffff954358788700 x1865013828495232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627278.626324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828495232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627278.626331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627278.626333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627278.626336:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627278.626341:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627278.626357:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627278.626364:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627278/real 1778627278] req@ffff95435878b100 x1865013828495360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627278.626385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828495360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627278.626390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627278.626392:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627278.626394:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627278.626399:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627278.626412:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627278.626418:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627278/real 1778627278] req@ffff95435878bb80 x1865013828495488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627278.626425:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828495488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627278.626430:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627278.626432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627278.626434:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627278.626447:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627278.626457:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627278.626462:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627278/real 1778627278] req@ffff95435878b480 x1865013828495616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627278.626476:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828495616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627278.626490:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627278.626493:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627278.626495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627278.817772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627278.817837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627279.585814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627279.841749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627279.841850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627280.609807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627280.865746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627280.865905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627281.633830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627281.889750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627281.889906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627282.657768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627282.913769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627282.913893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627283.681740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627283.745839:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627283.745848:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627283.745852:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627283.745860:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11618 00000100:00080000:1.0:1778627283.745886:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627283.745892:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627283.745921:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828495744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627283.745942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627283.745945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627283.745946:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627283.745949:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11618 00000100:00080000:1.0:1778627283.745951:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627283.745953:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627283.745961:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627283.745969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828495872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627283.745977:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627283.745980:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627283.745981:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627283.745983:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11618 00000100:00080000:1.0:1778627283.745985:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627283.745989:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828495744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627283.745990:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627283.745998:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627283.746011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828496000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627283.746021:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627283.746034:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627283.746036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627283.746041:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11618 00000100:00080000:1.0:1778627283.746048:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627283.746052:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627283.746058:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627283.746059:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627283.746075:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828496128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627283.746087:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627283.746090:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627283.746093:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627283/real 1778627283] req@ffff954350194380 x1865013828495744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627283.746094:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627283.746104:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11618 00000100:00080000:1.0:1778627283.746111:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627283.746117:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828495744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627283.746118:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627283.746127:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627283.746131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627283.746136:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627283.746139:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828496256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627283.746141:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828495744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627283.746169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627283.746185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627283.746193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627283/real 1778627283] req@ffff954350195880 x1865013828495872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627283.746218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828495872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627283.746231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627283.746233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627283.746236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828495872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627283.746240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627283.746255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627283.746262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627283/real 1778627283] req@ffff954350197b80 x1865013828496000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627283.746270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828496000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627283.746275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627283.746277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627283.746279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627283.746282:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627283.746292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627283.746297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627283/real 1778627283] req@ffff954350195180 x1865013828496128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627283.746304:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828496128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627283.746309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627283.746311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627283.746317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627283.746322:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627283.746333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627283.746339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627283/real 1778627283] req@ffff954350196a00 x1865013828496256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627283.746346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828496256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627283.746352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627283.746353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627283.746355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627283.937800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627283.937862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627284.705769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627284.961776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627284.961847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627285.729761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627285.985747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627285.985857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627286.753795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627287.009784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627287.009840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627287.777801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627288.033760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627288.033846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627288.801797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627288.865744:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627288.865754:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627288.865759:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627288.865769:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11623 00000100:00080000:0.0:1778627288.865782:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627288.865789:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627288.865830:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828496384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627288.865857:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627288.865879:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627288.865881:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627288.865885:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11623 00000100:00080000:0.0:1778627288.865889:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627288.865893:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627288.865904:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627288.865912:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828496384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627288.865917:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828496512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627288.865926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627288.865929:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627288.865930:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627288.865932:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11623 00000100:00080000:0.0:1778627288.865936:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627288.865938:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627288.865941:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627288.865951:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828496640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627288.865958:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627288.865960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627288.865962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627288.865966:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11623 00000100:00080000:0.0:1778627288.865969:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627288.865973:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627288.865987:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627288.865992:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627288.866007:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828496768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627288.866019:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627288.866022:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627288.866024:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627288.866028:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11623 00000100:00100000:2.0:1778627288.866032:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627288/real 1778627288] req@ffff954342f48e00 x1865013828496384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627288.866034:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627288.866041:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627288.866049:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627288.866060:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828496896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627288.866069:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828496384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627288.866083:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627288.866089:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627288.866094:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828496384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627288.866143:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627288.866164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627288.866174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627288/real 1778627288] req@ffff954342f4aa00 x1865013828496512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627288.866183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828496512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627288.866189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627288.866192:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627288.866194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627288.866199:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627288.866210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627288.866225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627288/real 1778627288] req@ffff954342f48380 x1865013828496640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627288.866233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828496640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627288.866238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627288.866241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627288.866243:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627288.866247:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627288.866259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627288.866265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627288/real 1778627288] req@ffff954342f48000 x1865013828496768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627288.866272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828496768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627288.866277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627288.866279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627288.866281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627288.866284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627288.866294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627288.866299:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627288/real 1778627288] req@ffff954342f49180 x1865013828496896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627288.866306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828496896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627288.866317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627288.866319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627288.866321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828496896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627289.057830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627289.057891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627289.825798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627290.081791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627290.081900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627290.849791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627291.105783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627291.105849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627291.873733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627292.129746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627292.129838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627292.897744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627293.153787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627293.153818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627293.921721:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627293.985737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627293.985745:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627293.985748:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627293.985757:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11628 00000100:00080000:0.0:1778627293.985769:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627293.985776:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627293.985813:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828497024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627293.985854:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627293.985857:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627293.985859:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627293.985862:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11628 00000100:00080000:0.0:1778627293.985866:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627293.985870:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627293.985880:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627293.985895:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828497152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627293.985900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828497024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627293.985915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627293.985918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627293.985920:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627293.985925:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11628 00000100:00080000:0.0:1778627293.985929:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627293.985932:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627293.985936:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627293.985954:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828497280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627293.985961:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627293.985964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627293.985965:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627293.985971:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11628 00000800:00000100:2.0:1778627293.985974:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627293.985981:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627293.985985:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627293.985989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627293.986014:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828497408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627293.986023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627293.986024:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627293/real 1778627293] req@ffff954342f4bb80 x1865013828497024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627293.986027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627293.986031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627293.986039:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11628 00000100:00080000:0.0:1778627293.986043:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627293.986047:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627293.986051:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627293.986053:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828497024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627293.986062:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828497536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627293.986063:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627293.986072:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627293.986079:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828497024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627293.986106:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627293.986130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627293.986144:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627293/real 1778627293] req@ffff954342f49880 x1865013828497152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627293.986155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828497152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627293.986162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627293.986164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627293.986166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627293.986177:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627293.986193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627293.986200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627293/real 1778627293] req@ffff954342f4a300 x1865013828497280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627293.986208:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828497280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627293.986213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627293.986216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627293.986218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627293.986221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627293.986241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627293.986247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627293/real 1778627293] req@ffff954342f4b480 x1865013828497408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627293.986255:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828497408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627293.986260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627293.986262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627293.986264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627293.986268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627293.986277:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627293.986283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627293/real 1778627293] req@ffff954342f49c00 x1865013828497536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627293.986295:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828497536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627293.986301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627293.986303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627293.986305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627294.177735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627294.177805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627294.945793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627295.201775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627295.201848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627295.969775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627296.225757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627296.225836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627296.993769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627297.249787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627297.249853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627298.017809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627298.273723:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627298.273870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627299.041759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627299.105833:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627299.105842:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627299.105853:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627299.105862:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11633 00000100:00080000:0.0:1778627299.105889:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627299.105895:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627299.105939:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828497664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627299.105962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627299.105965:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627299.105966:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627299.105970:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11633 00000100:00080000:0.0:1778627299.105972:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627299.105974:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627299.105981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627299.105990:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828497792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627299.105997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627299.105999:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627299.106000:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627299.106003:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11633 00000100:00080000:0.0:1778627299.106005:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627299.106007:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627299.106010:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627299.106017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828497920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627299.106017:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828497664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627299.106032:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627299.106038:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627299.106040:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627299.106043:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11633 00000100:00080000:0.0:1778627299.106046:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627299.106049:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627299.106053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627299.106063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828498048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627299.106069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627299.106071:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627299.106072:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627299.106075:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11633 00000100:00080000:0.0:1778627299.106078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627299.106080:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627299.106082:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627299.106091:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828498176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627299.106091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627299.106144:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627299/real 1778627299] req@ffff954342f49500 x1865013828497664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627299.106167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828497664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627299.106175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627299.106178:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627299.106181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828497664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627299.106187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627299.106206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627299.106214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627299/real 1778627299] req@ffff954342f4ad80 x1865013828497792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627299.106222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828497792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627299.106238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627299.106241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627299.106243:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627299.106264:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627299.106277:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627299.106284:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627299/real 1778627299] req@ffff954342f48a80 x1865013828497920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627299.106291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828497920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627299.106296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627299.106298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627299.106300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828497920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627299.106304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627299.106314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627299.106319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627299/real 1778627299] req@ffff954342f4b100 x1865013828498048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627299.106326:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828498048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627299.106330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627299.106332:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627299.106339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627299.106343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627299.106355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627299.106361:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627299/real 1778627299] req@ffff954342f49f80 x1865013828498176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627299.106369:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828498176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627299.106374:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627299.106376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627299.106378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627299.297782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627299.297811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627300.065810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627300.321772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627300.321795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627301.089741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627301.345753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627301.345788:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627302.113854:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627302.369804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627302.369877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627303.137788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627303.393801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627303.393887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627304.161850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627304.225746:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627304.225779:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627304.225784:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627304.225809:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11638 00000100:00080000:0.0:1778627304.225825:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627304.225834:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627304.225882:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828498304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627304.225918:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627304.225923:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627304.225925:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627304.225931:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11638 00000100:00080000:0.0:1778627304.225936:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627304.225940:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627304.225953:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627304.225969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828498432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627304.225981:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627304.225985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627304.225988:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627304.225991:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828498304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627304.225994:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11638 00000100:00080000:0.0:1778627304.226001:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627304.226006:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627304.226011:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627304.226024:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828498560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627304.226039:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627304.226042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627304.226043:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627304.226047:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11638 00000100:00080000:0.0:1778627304.226051:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627304.226054:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627304.226058:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627304.226069:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828498688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627304.226079:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627304.226083:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627304.226085:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627304.226088:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11638 00000100:00080000:0.0:1778627304.226093:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627304.226104:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627304.226110:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627304.226125:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828498816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627304.226132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627304.226163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627304.226179:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627304.226195:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627304.417764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627304.417855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627305.185840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627305.441755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627305.441871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627306.209801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627306.465792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627306.465851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627307.233768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627307.297933:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627307.297980:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627307.297999:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627307.298040:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627307.298051:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627307.298060:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627307.298069:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778627307.298104:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627304/real 1778627307] req@ffff954342f48a80 x1865013828498816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627307.298154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828498816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627307.298167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627307.298173:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627307.298178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627307.298186:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627304/real 1778627307] req@ffff954342f4a680 x1865013828498432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627307.298200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828498432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627307.298216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627307.298220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627307.298224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627307.298231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627304/real 1778627307] req@ffff954342f49f80 x1865013828498560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627307.298246:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828498560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627307.298254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627307.298258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627307.298262:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627307.298269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627304/real 1778627307] req@ffff954342f4b100 x1865013828498688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627307.298293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828498688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627307.298301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627307.298304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627307.298307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828498688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627307.298314:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627304/real 1778627307] req@ffff954342f48700 x1865013828498304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627307.298327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828498304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627307.298336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627307.298339:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627307.298343:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828498304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627307.489769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627307.489816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627308.257805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627308.513781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627308.513838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627309.281766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627309.345788:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627309.345797:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627309.345800:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627309.345808:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11643 00000100:00080000:2.0:1778627309.345818:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627309.345824:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627309.345852:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828498944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627309.345892:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627309.345895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627309.345903:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627309.345908:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11643 00000100:00080000:2.0:1778627309.345913:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627309.345918:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627309.345930:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627309.345943:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828498944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627309.345947:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828499072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627309.345960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627309.345963:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627309.345965:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627309.345969:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11643 00000100:00080000:2.0:1778627309.345973:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627309.345976:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627309.345981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627309.345992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828499200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627309.346001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627309.346004:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627309.346005:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627309.346011:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11643 00000800:00000100:0.0:1778627309.346013:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627309.346024:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627309.346029:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627309.346035:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627309.346048:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828499328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627309.346052:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627309/real 1778627309] req@ffff954358788380 x1865013828498944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627309.346069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627309.346075:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627309.346077:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627309.346082:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11643 00000100:00080000:2.0:1778627309.346086:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627309.346087:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828498944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627309.346092:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627309.346100:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627309.346109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778627309.346114:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828499456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627309.346122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627309.346131:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828498944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627309.346160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627309.346185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627309.346198:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627309/real 1778627309] req@ffff95435878aa00 x1865013828499072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627309.346212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828499072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627309.346222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627309.346233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627309.346237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627309.346242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627309.346263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627309.346274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627309/real 1778627309] req@ffff95435878b480 x1865013828499200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627309.346303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828499200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627309.346312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627309.346315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627309.346319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627309.346325:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627309.346349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627309.346361:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627309/real 1778627309] req@ffff95435878bb80 x1865013828499328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627309.346372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828499328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627309.346395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627309.346399:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627309.346402:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627309.346408:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627309.346432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627309.346443:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627309/real 1778627309] req@ffff95435878b100 x1865013828499456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627309.346454:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828499456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627309.346472:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627309.346476:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627309.346480:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627309.537748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627309.537807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627310.305777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627310.561771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627310.561834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627311.329746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627311.585728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627311.585799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627312.353766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627312.609781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627312.609857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627313.377776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627313.633743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627313.633792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627314.401753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627314.465744:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627314.465754:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627314.465759:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627314.465768:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11648 00000100:00080000:0.0:1778627314.465780:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627314.465787:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627314.465822:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828499584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627314.465854:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627314.465858:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627314.465860:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627314.465865:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11648 00000100:00080000:0.0:1778627314.465868:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627314.465872:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627314.465880:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627314.465891:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828499712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778627314.465896:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828499584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627314.465909:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627314.465913:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627314.465914:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627314.465918:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11648 00000100:00080000:0.0:1778627314.465930:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627314.465933:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627314.465937:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627314.465947:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828499840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627314.465953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627314.465956:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627314.465957:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627314.465960:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11648 00000100:00080000:0.0:1778627314.465963:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627314.465965:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627314.465968:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778627314.465974:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627314.465976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828499968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627314.465984:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627314.465987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627314.465989:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627314.465994:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11648 00000100:00080000:0.0:1778627314.466004:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627314.466009:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627314.466013:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627314/real 1778627314] req@ffff954342f4ad80 x1865013828499584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778627314.466025:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627314.466040:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828499584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627314.466041:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828500096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627314.466080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627314.466084:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627314.466089:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828499584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627314.466109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627314.466127:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627314.466137:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627314/real 1778627314] req@ffff954342f49500 x1865013828499712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627314.466145:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828499712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627314.466151:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627314.466153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627314.466155:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627314.466159:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627314.466169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627314.466184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627314/real 1778627314] req@ffff954342f49c00 x1865013828499840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627314.466192:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828499840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627314.466197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627314.466199:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627314.466201:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627314.466210:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627314.466225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627314.466231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627314/real 1778627314] req@ffff954342f4b480 x1865013828499968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627314.466239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828499968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627314.466244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627314.466246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627314.466248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828499968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627314.466251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627314.466262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627314.466267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627314/real 1778627314] req@ffff954342f4a300 x1865013828500096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627314.466274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828500096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627314.466279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627314.466281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627314.466282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627314.657729:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627314.657812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627315.425810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627315.681726:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627315.681791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627316.449775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627316.705771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627316.705826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627317.473783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627317.729776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627317.729826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627318.497767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627318.753765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627318.753852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627319.521805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627319.585799:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627319.585810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627319.585815:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627319.585830:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11654 00000100:00080000:0.0:1778627319.585867:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627319.585876:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627319.585951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828500224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627319.585978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627319.585983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627319.585986:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627319.585991:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11654 00000100:00080000:0.0:1778627319.585997:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627319.586006:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627319.586006:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828500224:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627319.586026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627319.586062:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828500352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627319.586072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627319.586075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627319.586076:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627319.586079:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11654 00000100:00080000:0.0:1778627319.586086:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627319.586089:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627319.586093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778627319.586096:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627319.586109:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828500480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627319.586120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627319.586124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627319.586126:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627319.586133:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11654 00000100:00100000:1.0:1778627319.586137:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627319/real 1778627319] req@ffff954342f49880 x1865013828500224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627319.586138:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627319.586146:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627319.586152:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627319.586179:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828500608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778627319.586183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828500224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627319.586241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627319.586241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627319.586253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627319.586258:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627319.586258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627319.586263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828500224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627319.586274:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11654 00000100:00080000:0.0:1778627319.586278:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627319.586281:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627319.586287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627319.586292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627319.586300:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828500736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778627319.586332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627319.586349:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627319/real 1778627319] req@ffff954342f4bb80 x1865013828500352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627319.586359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828500352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627319.586365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627319.586367:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627319.586370:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627319.586375:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627319.586389:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627319.586396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627319/real 1778627319] req@ffff954342f49180 x1865013828500480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627319.586403:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828500480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627319.586418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627319.586421:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627319.586423:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627319.586427:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627319.586441:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627319.586447:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627319/real 1778627319] req@ffff954342f48000 x1865013828500608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627319.586454:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828500608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627319.586459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627319.586461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627319.586463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627319.586475:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627319.586485:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627319.586491:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627319/real 1778627319] req@ffff954342f48380 x1865013828500736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627319.586517:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828500736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627319.586523:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627319.586525:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627319.586532:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627319.777745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627319.777904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627320.545744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627320.801776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627320.801849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627321.569809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627321.825786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627321.825875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627322.593735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627322.849747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627322.849824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627323.617750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627323.873731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627323.873853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627324.641739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627324.705750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627324.705757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627324.705761:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627324.705769:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11659 00000100:00080000:0.0:1778627324.705779:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627324.705785:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627324.705819:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828500864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627324.705859:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627324.705863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627324.705865:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627324.705869:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11659 00000100:00080000:0.0:1778627324.705872:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627324.705876:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627324.705898:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627324.705900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828500864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627324.705911:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828500992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627324.705919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627324.705921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627324.705923:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627324.705926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11659 00000100:00080000:0.0:1778627324.705929:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627324.705931:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627324.705935:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627324.705943:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828501120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627324.705949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627324.705952:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627324.705953:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627324.705956:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11659 00000100:00080000:0.0:1778627324.705958:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627324.705960:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778627324.705965:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778627324.705967:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627324.705976:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828501248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627324.705983:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627324.705986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627324.705991:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627324.705994:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11659 00000100:00080000:0.0:1778627324.705998:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778627324.705999:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627324/real 1778627324] req@ffff954342f4aa00 x1865013828500864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627324.706003:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627324.706012:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627324.706024:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828500864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627324.706025:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828501376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627324.706044:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627324.706051:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627324.706055:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828500864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627324.706074:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627324.706090:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627324.706097:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627324/real 1778627324] req@ffff954342f48e00 x1865013828500992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627324.706106:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828500992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627324.706112:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627324.706114:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627324.706116:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828500992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627324.706120:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627324.706129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627324.706144:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627324/real 1778627324] req@ffff954342f48380 x1865013828501120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627324.706157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828501120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627324.706165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627324.706168:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627324.706171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627324.706178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627324.706225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627324.706234:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627324/real 1778627324] req@ffff954342f48000 x1865013828501248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627324.706243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828501248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627324.706248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627324.706250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627324.706252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627324.706256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627324.706268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627324.706274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627324/real 1778627324] req@ffff954342f49180 x1865013828501376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627324.706281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828501376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627324.706291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627324.706294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627324.706296:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627324.897723:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627324.897807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627325.665757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627325.921820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627325.921881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627326.689765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627326.945770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627326.945842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627327.713832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627327.969784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627327.969852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627328.737805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627328.993776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627328.993846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627329.761787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627329.825782:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627329.825790:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627329.825794:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627329.825802:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11664 00000100:00080000:3.0:1778627329.825812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627329.825818:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627329.825851:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828501504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627329.825883:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627329.825887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627329.825889:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627329.825894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11664 00000100:00080000:3.0:1778627329.825909:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627329.825914:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627329.825924:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627329.825939:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828501504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627329.825941:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828501632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627329.825985:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627329.825987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627329.825989:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627329.825992:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11664 00000100:00080000:3.0:1778627329.825996:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627329.825999:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627329.826003:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627329.826013:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828501760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778627329.826021:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627329.826024:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627329.826027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627329.826029:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627329.826032:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11664 00000100:00080000:3.0:1778627329.826035:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627329.826037:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627329.826057:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627329.826077:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828501888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627329.826084:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627329.826087:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627329.826088:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778627329.826092:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627329/real 1778627329] req@ffff95434701ca80 x1865013828501504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627329.826093:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11664 00000100:00080000:3.0:1778627329.826105:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627329.826114:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627329.826121:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627329.826124:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828501504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627329.826152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778627329.826156:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828502016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627329.826160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627329.826166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828501504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627329.826204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627329.826224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627329.826233:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627329/real 1778627329] req@ffff95434701e680 x1865013828501632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627329.826242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828501632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627329.826254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627329.826257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627329.826259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627329.826270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627329.826284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627329.826304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627329/real 1778627329] req@ffff95434701f480 x1865013828501760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627329.826314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828501760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627329.826319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627329.826322:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627329.826324:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627329.826328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627329.826343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627329.826349:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627329/real 1778627329] req@ffff95434701df80 x1865013828501888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627329.826356:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828501888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627329.826362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627329.826364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627329.826366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828501888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627329.826369:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627329.826378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627329.826383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627329/real 1778627329] req@ffff95434701d180 x1865013828502016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627329.826396:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828502016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627329.826402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627329.826404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627329.826406:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627330.017791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627330.017852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627330.785728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627331.041738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627331.041807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627331.809765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627332.065803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627332.066022:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627332.833749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627333.089759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627333.089823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627333.857826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627334.113776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627334.113926:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627334.881720:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627334.945747:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627334.945757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627334.945762:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627334.945776:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11669 00000100:00080000:0.0:1778627334.945789:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627334.945798:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627334.945832:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828502144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627334.945859:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627334.945880:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627334.945882:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627334.945886:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11669 00000100:00080000:0.0:1778627334.945890:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627334.945897:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627334.945900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828502144:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627334.945917:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627334.945928:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828502272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627334.945936:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627334.945939:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627334.945940:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627334.945942:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11669 00000100:00080000:0.0:1778627334.945945:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627334.945947:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627334.945950:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627334.945956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828502400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627334.945963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627334.945965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627334.945967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627334.945969:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11669 00000100:00080000:0.0:1778627334.945971:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627334.945973:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778627334.945978:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778627334.945984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627334.945995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828502528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627334.946002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627334.946004:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627334.946006:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627334.946008:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11669 00000100:00080000:0.0:1778627334.946012:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627334.946015:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627334.946018:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627334.946023:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627334/real 1778627334] req@ffff954342f4bb80 x1865013828502144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627334.946027:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828502656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778627334.946066:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828502144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627334.946078:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627334.946083:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627334.946087:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828502144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627334.946109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627334.946131:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627334.946142:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627334/real 1778627334] req@ffff954342f49880 x1865013828502272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627334.946164:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828502272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627334.946187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627334.946207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627334.946211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627334.946218:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627334.946238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627334.946246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627334/real 1778627334] req@ffff954342f4a300 x1865013828502400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627334.946254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828502400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627334.946259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627334.946261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627334.946263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627334.946267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627334.946304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627334.946311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627334/real 1778627334] req@ffff954342f4b480 x1865013828502528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627334.946318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828502528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627334.946324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627334.946326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627334.946333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627334.946339:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627334.946377:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627334.946388:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627334/real 1778627334] req@ffff954342f49c00 x1865013828502656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627334.946400:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828502656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627334.946409:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627334.946413:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627334.946416:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627335.137755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627335.137827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627335.905744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627336.161783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627336.161827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627336.929771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627337.185760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627337.185886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627337.953833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627338.209829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627338.210182:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627338.977777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778627339.233920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627339.234002:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627340.001777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627340.065782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627340.065791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627340.065796:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627340.065804:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11674 00000100:00080000:2.0:1778627340.065815:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627340.065822:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627340.065858:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828502784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627340.065897:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627340.065901:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627340.065902:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627340.065906:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11674 00000100:00080000:2.0:1778627340.065939:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627340.065945:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627340.065954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627340.065963:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828502784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627340.065975:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828502912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627340.065988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627340.065992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627340.065993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627340.065998:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11674 00000100:00080000:2.0:1778627340.066002:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627340.066006:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627340.066010:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627340.066021:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828503040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627340.066029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627340.066033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627340.066034:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627340.066038:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11674 00000100:00080000:2.0:1778627340.066042:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627340.066045:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627340.066052:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627340.066062:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828503168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778627340.066073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627340.066109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627340.066113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627340.066127:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627340.066134:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11674 00000100:00080000:2.0:1778627340.066139:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627340.066143:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627340.066149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627340.066158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627340/real 1778627340] req@ffff954358788700 x1865013828502784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627340.066168:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828503296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778627340.066204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828502784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627340.066221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627340.066227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627340.066230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828502784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627340.066272:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627340.066310:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627340.066325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627340/real 1778627340] req@ffff954358789880 x1865013828502912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627340.066341:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828502912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627340.066363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627340.066367:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627340.066371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828502912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627340.066377:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778627340.066396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627340.066404:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627340/real 1778627340] req@ffff954358789c00 x1865013828503040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627340.066431:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828503040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627340.066438:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627340.066441:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627340.066443:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627340.066448:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627340.066461:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627340.066468:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627340/real 1778627340] req@ffff954358789500 x1865013828503168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627340.066483:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828503168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627340.066496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627340.066499:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627340.066501:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627340.066506:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778627340.066522:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627340.066529:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627340/real 1778627340] req@ffff954358788e00 x1865013828503296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627340.066537:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828503296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627340.066544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627340.066547:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627340.066549:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627340.257769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627340.257918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627341.025765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778627341.281827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627341.282007:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627342.049732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627342.305753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627342.305819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627343.073797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627343.329763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627343.329849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627344.097733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627344.353764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627344.353837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627345.121742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627345.185783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627345.185792:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627345.185796:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627345.185805:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11679 00000100:00080000:1.0:1778627345.185825:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627345.185849:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627345.185884:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828503424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627345.185918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627345.185921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627345.185922:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627345.185926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11679 00000100:00080000:1.0:1778627345.185930:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627345.185932:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627345.185941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627345.185953:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828503552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627345.185959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828503424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627345.185965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627345.185969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627345.185970:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627345.185974:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11679 00000100:00080000:1.0:1778627345.185976:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627345.185978:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627345.185981:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627345.186006:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828503680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627345.186016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627345.186020:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627345.186028:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627345.186033:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11679 00000800:00000100:0.0:1778627345.186038:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627345.186039:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627345.186045:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627345.186050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627345.186061:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828503808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627345.186068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627345.186070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627345.186072:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627345.186073:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627345/real 1778627345] req@ffff954350195f80 x1865013828503424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627345.186078:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11679 00000100:00080000:1.0:1778627345.186084:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627345.186087:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627345.186090:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627345.186101:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828503936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627345.186114:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828503424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627345.186124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627345.186127:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627345.186130:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828503424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627345.186149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627345.186166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627345.186173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627345/real 1778627345] req@ffff954350196a00 x1865013828503552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627345.186181:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828503552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627345.186187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627345.186189:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627345.186191:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627345.186194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627345.186204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627345.186209:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627345/real 1778627345] req@ffff954350195180 x1865013828503680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627345.186215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828503680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627345.186229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627345.186231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627345.186233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627345.186237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627345.186251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627345.186257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627345/real 1778627345] req@ffff954350197b80 x1865013828503808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627345.186263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828503808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627345.186268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627345.186270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627345.186272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627345.186276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627345.186285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627345.186290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627345/real 1778627345] req@ffff954350195880 x1865013828503936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627345.186297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828503936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627345.186302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627345.186304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627345.186312:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828503936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627345.377756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627345.377821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627346.145782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627346.401735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627346.401782:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627347.169754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627347.425754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627347.425875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627348.193827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627348.449784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627348.450004:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627349.217850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627349.473821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627349.473927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627350.241801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627350.305808:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627350.305819:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627350.305824:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627350.305834:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11684 00000100:00080000:3.0:1778627350.305845:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627350.305852:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627350.305903:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828504064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627350.305945:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627350.305948:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627350.305949:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627350.305953:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11684 00000100:00080000:3.0:1778627350.305956:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627350.305959:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627350.305966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627350.305977:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828504192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627350.305984:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627350.305987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627350.305988:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627350.305990:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11684 00000100:00080000:3.0:1778627350.305992:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627350.305997:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828504064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627350.305998:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627350.306004:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627350.306011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828504320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627350.306018:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627350.306021:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627350.306022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627350.306024:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11684 00000100:00080000:3.0:1778627350.306032:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627350.306035:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627350.306039:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627350.306047:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828504448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627350.306053:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627350.306058:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1778627350.306059:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627350.306061:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627350.306065:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11684 00000100:00080000:3.0:1778627350.306068:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627350.306071:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627350.306075:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627350.306084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828504576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627350.306093:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627350/real 1778627350] req@ffff95434701d880 x1865013828504064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627350.306110:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828504064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627350.306118:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627350.306121:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627350.306123:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828504064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627350.306129:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627350.306144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627350.306151:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627350/real 1778627350] req@ffff95434701ea00 x1865013828504192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627350.306159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828504192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627350.306164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627350.306167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627350.306175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627350.306195:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627350.306209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627350.306216:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627350/real 1778627350] req@ffff95434701c000 x1865013828504320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627350.306224:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828504320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627350.306229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627350.306230:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627350.306232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627350.306236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627350.306246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627350.306251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627350/real 1778627350] req@ffff95434701c700 x1865013828504448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627350.306259:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828504448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627350.306264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627350.306266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627350.306267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627350.306270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627350.306279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627350.306291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627350/real 1778627350] req@ffff95434701f800 x1865013828504576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627350.306298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828504576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627350.306304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627350.306306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627350.306308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627350.497798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627350.497856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627351.265756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627351.521770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627351.521830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627352.289736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627352.545749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627352.545822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627353.313747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627353.569765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627353.569880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627354.337787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627354.593749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627354.593790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627355.361751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627355.425774:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627355.425797:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627355.425802:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627355.425811:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11689 00000100:00080000:1.0:1778627355.425824:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627355.425831:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627355.425877:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828504704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627355.425922:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627355.425926:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627355.425928:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627355.425933:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11689 00000100:00080000:1.0:1778627355.425937:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627355.425942:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627355.425952:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627355.425968:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828504832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627355.425972:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828504704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627355.425986:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627355.425990:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627355.425991:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627355.425996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11689 00000100:00080000:1.0:1778627355.426000:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627355.426004:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627355.426009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627355.426021:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828504960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627355.426031:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627355.426035:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627355.426036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627355.426040:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11689 00000800:00000100:0.0:1778627355.426045:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627355.426053:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627355.426059:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627355.426065:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627355.426079:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828505088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627355.426086:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627355/real 1778627355] req@ffff954350194380 x1865013828504704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627355.426093:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627355.426098:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627355.426100:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627355.426104:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11689 00000100:00080000:1.0:1778627355.426109:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627355.426116:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627355.426121:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627355.426127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828504704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627355.426134:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828505216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627355.426145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627355.426153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627355.426158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828504704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627355.426185:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627355.426210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627355.426240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627355/real 1778627355] req@ffff954350194a80 x1865013828504832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627355.426250:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828504832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627355.426256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627355.426258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627355.426260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627355.426265:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627355.426279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627355.426285:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627355/real 1778627355] req@ffff954350195c00 x1865013828504960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627355.426293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828504960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627355.426298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627355.426299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627355.426301:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828504960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627355.426305:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627355.426315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627355.426320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627355/real 1778627355] req@ffff954350195500 x1865013828505088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627355.426327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828505088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627355.426337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627355.426340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627355.426342:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627355.426346:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627355.426358:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627355.426364:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627355/real 1778627355] req@ffff954350194e00 x1865013828505216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627355.426371:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828505216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627355.426377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627355.426380:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627355.426383:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627355.617773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627355.617839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627356.385742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627356.641741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627356.641796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627357.409776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627357.665794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627357.665857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627358.433774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627358.689782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627358.689839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627359.457774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627359.713771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627359.713831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627360.481760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627360.545770:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627360.545781:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627360.545801:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627360.545810:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11695 00000100:00080000:3.0:1778627360.545825:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627360.545831:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627360.545860:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828505344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627360.545882:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627360.545885:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627360.545887:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627360.545890:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11695 00000100:00080000:3.0:1778627360.545892:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627360.545895:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627360.545907:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627360.545917:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828505472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627360.545925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627360.545927:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627360.545928:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627360.545933:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11695 00000100:00100000:0.0:1778627360.545934:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828505344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627360.545935:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627360.545941:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627360.545945:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627360.545953:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828505600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627360.545969:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627360.545972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627360.545973:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627360.545976:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11695 00000100:00080000:3.0:1778627360.545979:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627360.545982:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627360.545986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627360.545993:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627360.546003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828505728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627360.546010:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627360.546013:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627360.546014:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627360.546017:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11695 00000100:00080000:3.0:1778627360.546020:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627360.546023:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627360/real 1778627360] req@ffff95434701f100 x1865013828505344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627360.546025:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627360.546031:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627360.546042:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828505856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627360.546047:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828505344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627360.546063:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627360.546067:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627360.546070:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828505344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627360.546088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627360.546104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627360.546112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627360/real 1778627360] req@ffff95434701c380 x1865013828505472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627360.546120:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828505472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627360.546125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627360.546128:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627360.546130:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627360.546133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627360.546141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627360.546146:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627360/real 1778627360] req@ffff95434701d500 x1865013828505600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627360.546153:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828505600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627360.546158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627360.546160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627360.546161:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627360.546170:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627360.546182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627360.546188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627360/real 1778627360] req@ffff95434701f800 x1865013828505728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627360.546195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828505728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627360.546200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627360.546202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627360.546204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627360.546207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627360.546216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627360.546220:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627360/real 1778627360] req@ffff95434701c700 x1865013828505856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627360.546227:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828505856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627360.546232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627360.546234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627360.546236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828505856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627360.737771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627360.737886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627361.505746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627361.761764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627361.761801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627362.529794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627362.785748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627362.785791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627363.553765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627363.809737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627363.809804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627364.577800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627364.833751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627364.833815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627365.601728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627365.665751:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627365.665759:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627365.665763:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627365.665771:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11700 00000100:00080000:3.0:1778627365.665780:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627365.665785:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627365.665823:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828505984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627365.665844:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627365.665847:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627365.665848:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627365.665852:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11700 00000100:00080000:3.0:1778627365.665854:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627365.665856:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627365.665863:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627365.665872:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828506112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627365.665879:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627365.665881:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627365.665883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627365.665885:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11700 00000100:00080000:3.0:1778627365.665887:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627365.665890:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627365.665892:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627365.665899:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828505984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627365.665915:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828506240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627365.665923:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627365.665926:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627365.665927:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627365.665930:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11700 00000100:00080000:3.0:1778627365.665935:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627365.665938:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627365.665944:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627365.665955:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627365.665958:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828506368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627365.665970:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627365.665974:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627365.665976:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627365.665981:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11700 00000100:00080000:3.0:1778627365.665985:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627365.665990:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627365.665995:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627365.665997:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627365/real 1778627365] req@ffff95434701c000 x1865013828505984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627365.666010:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828506496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627365.666015:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828505984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627365.666028:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627365.666032:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627365.666035:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828505984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627365.666041:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627365.666056:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627365.666063:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627365/real 1778627365] req@ffff95434701ea00 x1865013828506112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627365.666081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828506112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627365.666086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627365.666088:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627365.666091:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627365.666114:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627365.666125:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627365.666131:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627365/real 1778627365] req@ffff95434701d880 x1865013828506240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627365.666146:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828506240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627365.666152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627365.666154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627365.666156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627365.666165:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627365.666178:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627365.666184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627365/real 1778627365] req@ffff95434701d180 x1865013828506368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627365.666191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828506368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627365.666196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627365.666198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627365.666200:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627365.666204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627365.666212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627365.666217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627365/real 1778627365] req@ffff95434701df80 x1865013828506496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627365.666224:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828506496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627365.666238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627365.666240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627365.666242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627365.857749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627365.857815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627366.625804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627366.881740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627366.881845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627367.649757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627367.905740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627367.905796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627368.673774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627368.929773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627368.929856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627369.697746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627369.953736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627369.953793:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627370.721794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627370.785790:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627370.785798:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627370.785801:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627370.785809:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11705 00000100:00080000:1.0:1778627370.785819:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627370.785824:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627370.785855:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828506624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627370.785881:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627370.785899:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627370.785902:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627370.785908:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11705 00000100:00080000:1.0:1778627370.785913:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627370.785918:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627370.785927:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828506624:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627370.785928:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627370.785946:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828506752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627370.785954:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627370.785957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627370.785959:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627370.785961:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11705 00000100:00080000:1.0:1778627370.785964:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627370.785967:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627370.785970:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627370.785977:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828506880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627370.785984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627370.785986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627370.785988:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627370.785990:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11705 00000100:00080000:1.0:1778627370.785993:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627370.785995:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627370.786004:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627370.786012:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828507008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627370.786020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627370.786022:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627370.786024:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627370.786026:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11705 00000100:00080000:1.0:1778627370.786028:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627370.786031:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627370.786035:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627370.786036:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627370.786047:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828507136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627370.786076:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627370.786101:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627370.786127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627370.977771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627370.977833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627371.745786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627372.001733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627372.001794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627372.769762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627373.025782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627373.025853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627373.793737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627373.857943:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627373.857994:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627373.858012:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627373.858059:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627373.858070:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627373.858078:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627373.858085:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778627373.858176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627370/real 1778627373] req@ffff954350194000 x1865013828507136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627373.858201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828507136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627373.858215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627373.858235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627373.858240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627373.858248:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627370/real 1778627373] req@ffff954350194700 x1865013828506752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627373.858263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828506752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627373.858272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627373.858275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627373.858279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627373.858285:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627370/real 1778627373] req@ffff954350197480 x1865013828506880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627373.858297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828506880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627373.858305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627373.858309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627373.858312:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828506880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627373.858319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627370/real 1778627373] req@ffff954350196d80 x1865013828507008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627373.858330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828507008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627373.858338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627373.858347:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627373.858351:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627373.858358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627370/real 1778627373] req@ffff954350197800 x1865013828506624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627373.858371:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828506624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627373.858379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627373.858382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627373.858386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828506624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627374.049744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627374.049812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627374.817757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627375.073780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627375.073833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627375.841779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627375.905756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627375.905766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627375.905770:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627375.905778:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11710 00000100:00080000:1.0:1778627375.905788:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627375.905794:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627375.905822:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828507264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627375.905845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627375.905847:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627375.905849:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627375.905852:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11710 00000100:00080000:1.0:1778627375.905868:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627375.905872:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627375.905880:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627375.905888:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828507264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627375.905899:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828507392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627375.905913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627375.905917:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627375.905920:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627375.905924:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11710 00000100:00080000:1.0:1778627375.905929:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627375.905933:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627375.905938:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627375.905952:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828507520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627375.905958:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627375.905968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627375.905972:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627375.905974:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627375.905979:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11710 00000100:00080000:1.0:1778627375.905984:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627375.905989:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627375.905995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627375.905997:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627375/real 1778627375] req@ffff954350197100 x1865013828507264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627375.906009:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828507648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627375.906028:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828507264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627375.906051:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627375.906052:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627375.906057:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627375.906059:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627375.906065:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627375.906065:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828507264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627375.906076:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11710 00000100:00080000:1.0:1778627375.906082:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627375.906087:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627375.906093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627375.906098:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627375.906109:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828507776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627375.906130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627375.906156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627375/real 1778627375] req@ffff954350197800 x1865013828507392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627375.906180:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828507392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627375.906189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627375.906195:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627375.906199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627375.906205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627375.906227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627375.906238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627375/real 1778627375] req@ffff954350196d80 x1865013828507520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627375.906249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828507520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627375.906257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627375.906260:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627375.906263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627375.906269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627375.906289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627375.906299:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627375/real 1778627375] req@ffff954350197480 x1865013828507648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627375.906309:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828507648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627375.906324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627375.906328:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627375.906331:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627375.906349:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627375.906371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627375.906403:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627375/real 1778627375] req@ffff954350194700 x1865013828507776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627375.906415:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828507776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627375.906423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627375.906425:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627375.906428:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828507776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627376.097765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627376.097825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627376.865761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627377.121759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627377.121827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627377.889744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627378.145726:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627378.145769:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627378.913739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627379.169767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627379.169801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627379.937790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627380.193792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627380.193896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627380.961754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627381.025795:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627381.025804:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627381.025808:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627381.025815:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11715 00000100:00080000:1.0:1778627381.025825:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627381.025830:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627381.025868:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828507904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627381.025900:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627381.025909:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627381.025911:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627381.025915:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11715 00000100:00080000:1.0:1778627381.025918:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627381.025922:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627381.025943:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627381.025950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828507904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627381.025957:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828508032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627381.025966:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627381.025969:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627381.025970:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627381.025973:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11715 00000100:00080000:1.0:1778627381.025977:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627381.025980:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627381.025983:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627381.025991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828508160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627381.025997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627381.026000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627381.026001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627381.026006:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11715 00000800:00000100:2.0:1778627381.026007:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627381.026011:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627381.026015:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627381.026020:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627381.026027:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828508288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627381.026033:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627381.026036:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627381.026041:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627381.026046:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11715 00000100:00100000:2.0:1778627381.026046:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627381/real 1778627381] req@ffff954350194000 x1865013828507904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627381.026052:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627381.026057:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627381.026064:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627381.026072:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828508416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627381.026116:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828507904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627381.026126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627381.026130:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627381.026133:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828507904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627381.026152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627381.026170:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627381.026179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627381/real 1778627381] req@ffff954350194e00 x1865013828508032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627381.026187:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828508032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627381.026193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627381.026195:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627381.026198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627381.026201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627381.026218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627381.026228:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627381/real 1778627381] req@ffff954350195500 x1865013828508160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627381.026241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828508160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627381.026270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627381.026273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627381.026275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627381.026294:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627381.026318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627381.026330:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627381/real 1778627381] req@ffff954350195c00 x1865013828508288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627381.026343:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828508288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627381.026350:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627381.026352:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627381.026354:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627381.026360:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627381.026381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627381.026392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627381/real 1778627381] req@ffff954350194a80 x1865013828508416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627381.026404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828508416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627381.026413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627381.026416:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627381.026425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627381.217772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627381.217883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627381.985775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627382.241784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627382.241844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627383.009733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627383.265749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627383.265809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627384.033770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627384.289773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627384.289830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627385.057761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627385.313747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627385.313801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627386.081774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627386.145782:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627386.145793:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627386.145798:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627386.145807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11720 00000100:00080000:1.0:1778627386.145819:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627386.145826:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627386.145860:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828508544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627386.145886:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627386.145889:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627386.145890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627386.145893:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11720 00000100:00080000:1.0:1778627386.145896:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627386.145899:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627386.145905:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627386.145914:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828508672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627386.145933:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828508544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627386.145948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627386.145951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627386.145952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627386.145956:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11720 00000100:00080000:1.0:1778627386.145959:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627386.145962:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627386.145966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627386.145974:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828508800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627386.145981:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627386.145983:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627386.145984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627386.145986:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11720 00000100:00080000:1.0:1778627386.145989:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627386.145991:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627386.145994:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627386.145998:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627386.146008:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828508928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627386.146015:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627386.146018:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627386.146019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627386.146022:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11720 00000100:00080000:1.0:1778627386.146029:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627386.146033:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627386.146033:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627386/real 1778627386] req@ffff954350194380 x1865013828508544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627386.146037:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627386.146050:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828509056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627386.146058:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828508544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627386.146066:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627386.146069:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627386.146073:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828508544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627386.146093:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627386.146114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627386.146127:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627386/real 1778627386] req@ffff954350195880 x1865013828508672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627386.146140:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828508672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627386.146149:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627386.146153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627386.146165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627386.146171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627386.146209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627386.146221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627386/real 1778627386] req@ffff954350197b80 x1865013828508800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627386.146234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828508800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627386.146240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627386.146242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627386.146244:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627386.146249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627386.146263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627386.146270:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627386/real 1778627386] req@ffff954350195180 x1865013828508928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627386.146277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828508928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627386.146282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627386.146284:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627386.146285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828508928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627386.146289:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627386.146299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627386.146311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627386/real 1778627386] req@ffff954350196a00 x1865013828509056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627386.146319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828509056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627386.146324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627386.146326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627386.146328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627386.337766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627386.337834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627387.105741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627387.361757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627387.361812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627388.129758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627388.385759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627388.385814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627389.153750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627389.409745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627389.409796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627390.177831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627390.433801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627390.433872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627391.201733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627391.265773:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627391.265781:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627391.265785:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627391.265793:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11725 00000100:00080000:1.0:1778627391.265803:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627391.265808:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627391.265840:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828509184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627391.265863:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627391.265866:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627391.265867:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627391.265870:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11725 00000100:00080000:1.0:1778627391.265873:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627391.265875:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627391.265881:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627391.265891:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828509312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627391.265898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627391.265914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627391.265916:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627391.265918:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11725 00000100:00080000:1.0:1778627391.265921:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627391.265925:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627391.265929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627391.265939:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828509440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627391.265946:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828509184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627391.265951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627391.265961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627391.265963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627391.265967:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11725 00000100:00080000:1.0:1778627391.265969:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627391.265971:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627391.265974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627391.265985:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828509568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627391.265991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627391.265994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627391.265996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627391.265998:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11725 00000100:00080000:1.0:1778627391.266000:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627391.266002:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627391.266010:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627391.266019:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828509696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627391.266051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627391.266107:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627391/real 1778627391] req@ffff954350195f80 x1865013828509184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627391.266125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828509184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627391.266135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627391.266138:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627391.266142:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828509184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627391.266147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627391.266164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627391.266188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627391/real 1778627391] req@ffff954350196a00 x1865013828509312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627391.266196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828509312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627391.266203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627391.266205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627391.266207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627391.266230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627391.266242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627391.266248:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627391/real 1778627391] req@ffff954350195180 x1865013828509440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627391.266256:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828509440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627391.266260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627391.266262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627391.266264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627391.266268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627391.266279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627391.266284:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627391/real 1778627391] req@ffff954350197b80 x1865013828509568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627391.266291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828509568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627391.266301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627391.266304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627391.266307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627391.266311:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627391.266324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627391.266330:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627391/real 1778627391] req@ffff954350195880 x1865013828509696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627391.266338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828509696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627391.266344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627391.266346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627391.266348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627391.457785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627391.457790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627392.225757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627392.481810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627392.481900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627393.249733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627393.505800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627393.505832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627394.273763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627394.529786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627394.529868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627395.297820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627395.553809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627395.553822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627396.321749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627396.385755:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627396.385763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627396.385766:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627396.385775:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11730 00000100:00080000:1.0:1778627396.385784:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627396.385791:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627396.385821:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828509824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627396.385851:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627396.385854:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627396.385855:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627396.385858:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11730 00000100:00080000:1.0:1778627396.385861:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627396.385863:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627396.385870:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627396.385879:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828509952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627396.385886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627396.385888:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627396.385890:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627396.385892:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11730 00000100:00080000:1.0:1778627396.385907:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627396.385910:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627396.385915:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627396.385929:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828510080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627396.385935:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828509824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627396.385955:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627396.385958:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627396.385961:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627396.385968:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11730 00000100:00080000:1.0:1778627396.385973:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627396.385989:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627396.385996:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627396.386007:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828510208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627396.386015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627396.386017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627396.386018:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627396.386021:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11730 00000100:00080000:1.0:1778627396.386025:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627396.386028:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627396.386033:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627396.386041:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828510336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627396.386056:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627396.386111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627396/real 1778627396] req@ffff954350194380 x1865013828509824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627396.386136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828509824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627396.386158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627396.386161:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627396.386165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828509824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627396.386171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627396.386193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627396.386203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627396/real 1778627396] req@ffff954350194a80 x1865013828509952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627396.386211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828509952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627396.386217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627396.386220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627396.386222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828509952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627396.386251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627396.386263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627396.386269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627396/real 1778627396] req@ffff954350195c00 x1865013828510080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627396.386277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828510080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627396.386282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627396.386285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627396.386287:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627396.386296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627396.386311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627396.386318:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627396/real 1778627396] req@ffff954350195500 x1865013828510208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627396.386326:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828510208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627396.386331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627396.386333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627396.386335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627396.386339:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627396.386349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627396.386354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627396/real 1778627396] req@ffff954350194e00 x1865013828510336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627396.386362:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828510336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627396.386367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627396.386369:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627396.386371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627396.577799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627396.577840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627397.345772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627397.601798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627397.601890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627398.369748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627398.625766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627398.625828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627399.393811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627399.649762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627399.649829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627400.417724:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627400.673775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627400.673839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627401.441767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627401.505759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627401.505770:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627401.505775:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627401.505785:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11736 00000100:00080000:1.0:1778627401.505798:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627401.505805:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627401.505838:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828510464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627401.505862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627401.505866:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627401.505867:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627401.505870:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11736 00000100:00080000:1.0:1778627401.505872:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627401.505887:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627401.505896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627401.505907:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828510592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627401.505935:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627401.505935:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828510464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627401.505945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627401.505963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627401.505967:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11736 00000100:00080000:1.0:1778627401.505972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627401.505976:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627401.505981:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627401.505992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828510720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627401.506000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778627401.506006:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627401.506007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627401.506011:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627401.506016:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11736 00000100:00080000:1.0:1778627401.506021:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627401.506025:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627401.506029:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627401.506039:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828510848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627401.506047:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627401/real 1778627401] req@ffff954350194000 x1865013828510464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627401.506063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627401.506070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627401.506071:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627401.506076:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11736 00000100:00080000:1.0:1778627401.506079:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627401.506081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828510464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627401.506085:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627401.506094:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627401.506099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627401.506109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627401.506113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828510464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627401.506118:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828510976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627401.506146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627401.506166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627401.506176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627401/real 1778627401] req@ffff954350194700 x1865013828510592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627401.506186:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828510592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627401.506192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627401.506195:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627401.506198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627401.506202:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627401.506230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627401.506238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627401/real 1778627401] req@ffff954350197480 x1865013828510720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627401.506258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828510720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627401.506265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627401.506268:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627401.506271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627401.506276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627401.506292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627401.506300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627401/real 1778627401] req@ffff954350196d80 x1865013828510848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627401.506310:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828510848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627401.506316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627401.506318:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627401.506321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627401.506325:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627401.506337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627401.506344:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627401/real 1778627401] req@ffff954350197800 x1865013828510976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627401.506353:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828510976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627401.506371:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627401.506374:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627401.506377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828510976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627401.697769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627401.697831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627402.465739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627402.721798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627402.721891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627403.489783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627403.745779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627403.746022:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627404.513730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627404.769765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627404.770489:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627405.537732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627405.793788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627405.794288:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627406.561720:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627406.625767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627406.625788:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627406.625791:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627406.625800:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11741 00000100:00080000:1.0:1778627406.625809:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627406.625815:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627406.625846:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828511104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627406.625870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627406.625873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627406.625874:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627406.625877:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11741 00000100:00080000:1.0:1778627406.625879:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627406.625882:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627406.625889:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627406.625904:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828511232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627406.625911:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627406.625914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627406.625915:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627406.625931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11741 00000100:00080000:1.0:1778627406.625934:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627406.625937:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627406.625941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627406.625946:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828511104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627406.625951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828511360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627406.625959:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627406.625961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627406.625962:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627406.625964:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11741 00000100:00080000:1.0:1778627406.625966:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627406.625968:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627406.625971:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627406.625979:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828511488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627406.625985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627406.625987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627406.625989:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627406.625991:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11741 00000100:00080000:1.0:1778627406.626000:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627406.626004:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627406.626009:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627406.626010:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627406.626020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828511616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627406.626044:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627406/real 1778627406] req@ffff954350197100 x1865013828511104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627406.626059:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828511104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627406.626067:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627406.626070:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627406.626073:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828511104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627406.626077:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627406.626091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627406.626097:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627406/real 1778627406] req@ffff954350197800 x1865013828511232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627406.626105:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828511232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627406.626119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627406.626122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627406.626124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627406.626141:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627406.626154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627406.626161:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627406/real 1778627406] req@ffff954350196d80 x1865013828511360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627406.626168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828511360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627406.626173:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627406.626174:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627406.626176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627406.626180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627406.626189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627406.626194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627406/real 1778627406] req@ffff954350197480 x1865013828511488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627406.626202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828511488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627406.626206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627406.626208:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627406.626214:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627406.626218:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627406.626230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627406.626236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627406/real 1778627406] req@ffff954350194700 x1865013828511616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627406.626243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828511616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627406.626249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627406.626251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627406.626252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627406.817809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627406.817927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627407.585741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627407.841805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627407.842037:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627408.609752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627408.865743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627408.865807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627409.633777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627409.889743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627409.889837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627410.657817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627410.913769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627410.913833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627411.681765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627411.745756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627411.745763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627411.745767:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627411.745774:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11746 00000100:00080000:1.0:1778627411.745784:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627411.745790:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627411.745820:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828511744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627411.745845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627411.745848:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627411.745849:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627411.745852:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11746 00000100:00080000:1.0:1778627411.745855:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627411.745857:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627411.745864:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627411.745874:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828511872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627411.745881:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627411.745884:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627411.745885:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627411.745888:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11746 00000100:00080000:1.0:1778627411.745890:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627411.745892:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627411.745900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828511744:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627411.745910:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627411.745919:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828512000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627411.745927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627411.745930:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627411.745931:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627411.745934:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11746 00000100:00080000:1.0:1778627411.745937:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627411.745940:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627411.745943:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627411.745950:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828512128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627411.745956:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778627411.745958:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627411.745961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627411.745963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627411.745966:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11746 00000100:00080000:1.0:1778627411.745968:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627411.745971:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627411.745974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627411.745989:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828512256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627411.745999:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627411/real 1778627411] req@ffff954350194000 x1865013828511744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627411.746015:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828511744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627411.746024:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627411.746028:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627411.746037:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828511744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627411.746042:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627411.746059:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627411.746067:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627411/real 1778627411] req@ffff954350194e00 x1865013828511872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627411.746076:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828511872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627411.746081:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627411.746084:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627411.746086:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828511872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627411.746103:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627411.746114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627411.746119:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627411/real 1778627411] req@ffff954350195500 x1865013828512000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627411.746126:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828512000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627411.746131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627411.746133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627411.746135:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627411.746139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627411.746148:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627411.746160:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627411/real 1778627411] req@ffff954350195c00 x1865013828512128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627411.746168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828512128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627411.746173:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627411.746175:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627411.746176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627411.746180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627411.746191:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627411.746196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627411/real 1778627411] req@ffff954350194a80 x1865013828512256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627411.746203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828512256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627411.746208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627411.746210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627411.746211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627411.937757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627411.937820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627412.705786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627412.961743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627412.961805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627413.729745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627413.985753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627413.985810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627414.753794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627415.009742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627415.009784:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627415.777755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627416.033766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627416.033833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627416.801756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627416.865763:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627416.865773:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627416.865794:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627416.865806:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11751 00000100:00080000:1.0:1778627416.865818:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627416.865826:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627416.865857:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828512384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627416.865883:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627416.865887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627416.865889:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627416.865893:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11751 00000100:00080000:1.0:1778627416.865897:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627416.865902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627416.865911:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627416.865924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828512512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627416.865933:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828512384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627416.865934:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627416.865941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627416.865944:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627416.865948:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11751 00000100:00080000:1.0:1778627416.865950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627416.865953:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627416.865956:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627416.865966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828512640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627416.865983:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627416.865986:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627416.865987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627416.865991:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11751 00000100:00080000:1.0:1778627416.865994:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627416.865996:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627416.866000:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627416.866004:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627416.866013:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828512768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627416.866020:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627416.866023:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627416.866025:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627416.866027:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11751 00000100:00080000:1.0:1778627416.866031:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627416.866034:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627416.866037:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627416.866046:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828512896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627416.866051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627416/real 1778627416] req@ffff954350194380 x1865013828512384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627416.866074:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828512384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627416.866086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627416.866091:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627416.866095:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828512384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627416.866119:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627416.866143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627416.866157:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627416/real 1778627416] req@ffff954350195880 x1865013828512512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627416.866170:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828512512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627416.866179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627416.866182:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627416.866186:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627416.866192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627416.866211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627416.866221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627416/real 1778627416] req@ffff954350197b80 x1865013828512640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627416.866234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828512640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627416.866256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627416.866260:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627416.866264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627416.866271:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627416.866293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627416.866303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627416/real 1778627416] req@ffff954350195180 x1865013828512768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627416.866315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828512768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627416.866323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627416.866326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627416.866330:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627416.866335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627416.866349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627416.866357:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627416/real 1778627416] req@ffff954350196a00 x1865013828512896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627416.866368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828512896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627416.866377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627416.866379:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627416.866386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828512896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627417.057744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627417.057800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627417.825743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627418.081766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627418.081828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627418.849728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627419.105763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627419.105823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627419.873748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627420.129761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627420.129808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627420.897734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627421.153755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627421.153823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627421.921742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627421.985802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627421.985813:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627421.985818:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627421.985828:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11756 00000100:00080000:2.0:1778627421.985840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627421.985847:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627421.985886:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828513024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627421.985924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627421.985928:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627421.985930:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627421.985935:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11756 00000100:00080000:2.0:1778627421.985939:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627421.985943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627421.985953:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627421.985967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828513152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627421.985977:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627421.985994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627421.985996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627421.985999:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11756 00000100:00080000:2.0:1778627421.986002:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627421.986005:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627421.986009:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627421.986020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828513280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627421.986030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627421.986033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627421.986035:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627421.986038:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11756 00000100:00080000:2.0:1778627421.986041:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627421.986043:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627421.986047:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627421.986054:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828513408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627421.986060:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627421.986062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627421.986063:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627421.986065:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11756 00000100:00080000:2.0:1778627421.986067:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627421.986068:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627421.986076:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627421.986083:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828513536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627421.986158:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828513024:192.168.204.102@tcp:250:kworker.0 00000800:00000100:0.0:1778627421.986226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627421.986259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627421/real 1778627421] req@ffff954358789180 x1865013828513024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627421.986278:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828513024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627421.986285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627421.986288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627421.986291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828513024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627421.986296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627421.986313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627421.986321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627421/real 1778627421] req@ffff954358788a80 x1865013828513152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627421.986328:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828513152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627421.986334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627421.986335:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627421.986337:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627421.986341:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627421.986351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627421.986356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627421/real 1778627421] req@ffff95435878b800 x1865013828513280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627421.986373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828513280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627421.986390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627421.986393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627421.986397:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627421.986403:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627421.986425:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627421.986438:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627421/real 1778627421] req@ffff95435878a300 x1865013828513408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627421.986450:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828513408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627421.986458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627421.986461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627421.986464:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627421.986483:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627421.986501:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627421.986508:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627421/real 1778627421] req@ffff95435878ad80 x1865013828513536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627421.986516:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828513536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627421.986531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627421.986533:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627421.986535:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627422.177756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627422.177810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627422.945762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627423.201758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627423.201918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627423.969757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627424.225755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627424.225781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627424.993778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627425.249765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627425.249802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627426.017759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627426.273759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627426.273815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627427.041777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627427.105810:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627427.105821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627427.105825:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627427.105835:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11761 00000100:00080000:1.0:1778627427.105864:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627427.105871:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627427.105904:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828513664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627427.105933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627427.105937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627427.105939:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627427.105944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11761 00000100:00080000:1.0:1778627427.105949:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627427.105954:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627427.105964:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627427.105976:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828513664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627427.105979:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828513792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627427.105993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627427.105997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627427.105999:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627427.106003:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11761 00000100:00080000:1.0:1778627427.106007:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627427.106011:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627427.106016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627427.106029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828513920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627427.106036:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627427.106047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627427.106056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627427.106059:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627427.106065:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11761 00000100:00080000:1.0:1778627427.106070:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627427.106072:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627427/real 1778627427] req@ffff954350195f80 x1865013828513664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627427.106079:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627427.106090:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627427.106094:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828513664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627427.106103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778627427.106108:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828514048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627427.106111:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627427.106120:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828513664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627427.106132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627427.106135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627427.106136:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627427.106140:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11761 00000100:00080000:1.0:1778627427.106143:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627427.106147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513792:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627427.106150:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627427.106159:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627427.106173:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828514176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627427.106182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627427.106198:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627427/real 1778627427] req@ffff954350196a00 x1865013828513792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627427.106220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828513792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627427.106227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627427.106229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627427.106231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627427.106235:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627427.106249:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627427.106256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627427/real 1778627427] req@ffff954350195180 x1865013828513920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627427.106263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828513920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627427.106268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627427.106270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627427.106271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828513920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627427.106275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627427.106285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627427.106290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627427/real 1778627427] req@ffff954350197b80 x1865013828514048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627427.106297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828514048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627427.106310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627427.106312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627427.106315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627427.106327:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627427.106359:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627427.106383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627427/real 1778627427] req@ffff954350195880 x1865013828514176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627427.106396:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828514176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627427.106402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627427.106405:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627427.106407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627427.297767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627427.297828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627428.065770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627428.321749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627428.321796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627429.089779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627429.345764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627429.345821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627430.113799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627430.369768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627430.369837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627431.137791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627431.393773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627431.393922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627432.161747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627432.225758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627432.225765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627432.225769:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627432.225777:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11766 00000100:00080000:2.0:1778627432.225786:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627432.225791:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627432.225829:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828514304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627432.225859:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627432.225863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627432.225865:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627432.225870:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11766 00000100:00080000:2.0:1778627432.225875:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627432.225893:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828514304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627432.225897:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627432.225907:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627432.225923:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828514432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627432.225935:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627432.225939:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627432.225941:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627432.225945:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11766 00000800:00000100:0.0:1778627432.225949:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627432.225951:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627432.225957:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627432.225962:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627432.225974:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828514560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627432.225983:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627432/real 1778627432] req@ffff95435878a680 x1865013828514304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627432.225986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627432.225994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627432.225997:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627432.226004:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11766 00000100:00080000:2.0:1778627432.226008:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627432.226011:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828514304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627432.226012:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627432.226016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627432.226025:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828514688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627432.226049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627432.226052:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627432.226057:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778627432.226060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627432.226063:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828514304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627432.226064:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627432.226073:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11766 00000100:00080000:2.0:1778627432.226076:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627432.226079:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627432.226083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627432.226090:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627432.226093:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828514816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627432.226117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627432.226127:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627432/real 1778627432] req@ffff954358789f80 x1865013828514432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627432.226136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828514432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627432.226142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627432.226144:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627432.226146:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627432.226150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627432.226160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627432.226165:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627432/real 1778627432] req@ffff95435878ad80 x1865013828514560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627432.226172:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828514560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627432.226177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627432.226179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627432.226181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627432.226189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627432.226202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627432.226208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627432/real 1778627432] req@ffff95435878a300 x1865013828514688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627432.226216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828514688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627432.226221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627432.226223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627432.226224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627432.226236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627432.226253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627432.226259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627432/real 1778627432] req@ffff95435878b800 x1865013828514816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627432.226280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828514816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627432.226288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627432.226290:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627432.226292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828514816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627432.417731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627432.417822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627433.185776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627433.441726:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627433.441799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627434.209787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627434.465750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627434.465806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627435.233746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627435.489774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627435.489840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627436.257814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627436.513759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627436.513831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627437.281763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627437.345805:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627437.345812:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627437.345816:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627437.345825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11771 00000100:00080000:1.0:1778627437.345835:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627437.345840:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627437.345873:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828514944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627437.345896:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627437.345898:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627437.345900:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627437.345903:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11771 00000100:00080000:1.0:1778627437.345905:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627437.345907:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627437.345914:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627437.345923:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828515072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627437.345929:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627437.345932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627437.345933:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627437.345936:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11771 00000100:00080000:1.0:1778627437.345937:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627437.345942:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627437.345943:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828514944:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627437.345960:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627437.345970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828515200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627437.345978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627437.345980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627437.345981:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627437.345983:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11771 00000100:00080000:1.0:1778627437.345988:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627437.345991:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627437.345994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627437.346002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828515328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627437.346009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627437.346011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627437.346013:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627437.346016:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11771 00000100:00080000:1.0:1778627437.346019:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627437.346022:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627437.346026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627437.346029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627437.346035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828515456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627437.346066:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627437.346083:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627437.346098:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627437.537765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627437.537836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627438.305791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627438.561760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627438.561823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627439.329764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627439.585762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627439.585813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627440.353754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627440.417920:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627440.417966:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627440.417983:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627440.418021:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627440.418031:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627440.418040:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627440.418051:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627440.418083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627437/real 1778627440] req@ffff954350195c00 x1865013828515200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627440.418120:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828515200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627440.418132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627440.418138:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627440.418142:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627440.418150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627437/real 1778627440] req@ffff954350195500 x1865013828515328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627440.418162:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828515328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627440.418169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627440.418187:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627440.418190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627440.418197:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627437/real 1778627440] req@ffff954350194e00 x1865013828515456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627440.418216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828515456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627440.418224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627440.418227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627440.418230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627440.418236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627437/real 1778627440] req@ffff954350194380 x1865013828514944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627440.418254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828514944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627440.418262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627440.418265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627440.418269:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828514944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627440.418276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627437/real 1778627440] req@ffff954350194a80 x1865013828515072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627440.418288:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828515072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627440.418297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627440.418308:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627440.418312:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627440.609770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627440.609817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627441.377754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627441.633754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627441.633819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627442.401767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627442.465757:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627442.465768:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627442.465773:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627442.465783:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11776 00000100:00080000:0.0:1778627442.465794:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627442.465814:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627442.465848:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828515584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627442.465893:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627442.465897:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627442.465899:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627442.465903:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11776 00000100:00080000:0.0:1778627442.465907:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627442.465913:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627442.465924:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627442.465935:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828515584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627442.465942:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828515712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627442.465956:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627442.465960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627442.465962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627442.465981:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11776 00000100:00080000:0.0:1778627442.465986:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627442.465991:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627442.466007:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627442.466021:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828515840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627442.466031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627442.466037:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627442.466042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627442.466044:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627442.466050:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11776 00000100:00080000:0.0:1778627442.466054:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627442.466059:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627442.466079:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627442.466086:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627442/real 1778627442] req@ffff954342f49500 x1865013828515584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627442.466106:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828515968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627442.466120:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627442.466126:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778627442.466127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828515584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627442.466129:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627442.466139:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11776 00000100:00080000:0.0:1778627442.466146:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627442.466147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627442.466151:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627442.466157:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:0.0:1778627442.466159:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627442.466183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828515584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627442.466187:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828516096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627442.466219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627442.466260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627442.466274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627442/real 1778627442] req@ffff954342f4ad80 x1865013828515712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627442.466287:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828515712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627442.466296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627442.466299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627442.466302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627442.466308:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627442.466326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627442.466336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627442/real 1778627442] req@ffff954342f48700 x1865013828515840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627442.466348:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828515840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627442.466356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627442.466359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627442.466362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627442.466376:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627442.466398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627442.466419:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627442/real 1778627442] req@ffff954342f4b100 x1865013828515968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627442.466431:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828515968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627442.466439:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627442.466442:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627442.466446:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828515968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627442.466451:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627442.466469:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627442.466480:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627442/real 1778627442] req@ffff954342f49f80 x1865013828516096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627442.466491:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828516096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627442.466498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627442.466501:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627442.466503:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627442.657776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627442.657839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627443.425751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627443.681763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627443.681819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627444.449786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627444.705779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627444.705867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627445.473738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627445.729753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627445.729875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627446.497757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627446.753776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627446.753836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627447.521836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627447.585798:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627447.585809:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627447.585814:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627447.585824:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11782 00000100:00080000:0.0:1778627447.585837:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627447.585844:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627447.585886:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828516224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627447.585912:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627447.585915:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627447.585916:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627447.585920:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11782 00000100:00080000:0.0:1778627447.585922:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627447.585924:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627447.585932:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627447.585941:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828516352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627447.585948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627447.585951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627447.585986:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627447.585992:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11782 00000100:00080000:0.0:1778627447.585998:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627447.586007:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627447.586008:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828516224:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627447.586015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627447.586035:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828516480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627447.586043:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627447.586045:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627447.586047:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627447.586051:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11782 00000100:00080000:0.0:1778627447.586056:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627447.586061:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627447.586067:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627447.586083:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828516608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627447.586094:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627447.586097:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627447.586100:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627447.586109:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11782 00000800:00000100:2.0:1778627447.586111:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627447.586116:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627447.586120:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627447.586125:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627447.586150:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828516736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627447.586178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627447/real 1778627447] req@ffff954342f4a680 x1865013828516224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627447.586202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828516224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627447.586215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627447.586220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627447.586224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828516224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627447.586233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627447.586261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627447.586275:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627447/real 1778627447] req@ffff954342f48a80 x1865013828516352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627447.586289:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828516352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627447.586298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627447.586302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627447.586305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627447.586332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627447.586354:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627447.586366:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627447/real 1778627447] req@ffff954342f49f80 x1865013828516480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627447.586379:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828516480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627447.586403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627447.586407:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627447.586411:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627447.586418:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627447.586439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627447.586450:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627447/real 1778627447] req@ffff954342f4b100 x1865013828516608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627447.586462:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828516608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627447.586470:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627447.586473:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627447.586476:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627447.586483:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627447.586510:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627447.586521:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627447/real 1778627447] req@ffff954342f48700 x1865013828516736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627447.586534:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828516736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627447.586543:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627447.586547:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627447.586555:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627447.777773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627447.777850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627448.545827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627448.801779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627448.801976:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627449.569754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627449.825760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627449.825834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627450.593801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627450.849740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627450.849864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627451.617816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627451.873716:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627451.873822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627452.641792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627452.705801:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627452.705808:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627452.705812:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627452.705822:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11787 00000100:00080000:0.0:1778627452.705835:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627452.705842:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627452.705891:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828516864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627452.705916:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627452.705919:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627452.705920:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627452.705924:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11787 00000100:00080000:0.0:1778627452.705927:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627452.705930:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627452.705938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627452.705950:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828516992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627452.705962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627452.705965:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828516864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627452.705967:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627452.705984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627452.705990:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11787 00000100:00080000:0.0:1778627452.705996:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627452.706001:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627452.706006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778627452.706021:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627452.706022:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828517120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627452.706030:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627452.706033:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627452.706035:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627452.706038:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11787 00000100:00080000:0.0:1778627452.706041:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627452.706045:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627452.706049:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627452.706055:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627452/real 1778627452] req@ffff954342f4ad80 x1865013828516864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627452.706060:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828517248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627452.706074:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627452.706077:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627452.706078:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627452.706081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828516864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627452.706084:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11787 00000100:00080000:0.0:1778627452.706090:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627452.706096:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627452.706099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778627452.706100:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627452.706104:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627452.706110:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828516864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627452.706118:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828517376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627452.706133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627452.706151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627452.706163:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627452/real 1778627452] req@ffff954342f49500 x1865013828516992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627452.706171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828516992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627452.706177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627452.706185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627452.706188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828516992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627452.706192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627452.706206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627452.706212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627452/real 1778627452] req@ffff954342f49c00 x1865013828517120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627452.706219:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828517120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627452.706224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627452.706226:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627452.706228:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627452.706231:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627452.706242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627452.706247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627452/real 1778627452] req@ffff954342f4b480 x1865013828517248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627452.706254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828517248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627452.706259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627452.706261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627452.706262:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627452.706266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627452.706281:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627452.706291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627452/real 1778627452] req@ffff954342f4a300 x1865013828517376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627452.706304:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828517376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627452.706313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627452.706317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627452.706320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627452.897751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627452.897828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627453.665804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627453.921749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627453.921811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627454.689744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627454.945738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627454.945809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627455.713788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627455.969746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627455.969828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627456.737755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627456.993795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627456.993845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627457.761782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627457.825756:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627457.825776:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627457.825780:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627457.825788:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11792 00000100:00080000:1.0:1778627457.825797:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627457.825803:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627457.825836:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828517504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627457.825856:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627457.825859:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627457.825860:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627457.825863:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11792 00000100:00080000:1.0:1778627457.825879:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627457.825882:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627457.825890:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627457.825895:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828517504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627457.825903:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828517632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627457.825911:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627457.825914:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627457.825916:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627457.825920:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11792 00000100:00080000:1.0:1778627457.825925:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627457.825929:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627457.825934:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627457.825946:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828517760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627457.825956:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627457.825963:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627457.825968:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627457.825970:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627457.825976:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11792 00000100:00100000:2.0:1778627457.825988:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627457/real 1778627457] req@ffff954350194000 x1865013828517504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627457.825990:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627457.826000:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627457.826006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627457.826009:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828517504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627457.826024:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828517888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627457.826047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627457.826052:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627457.826057:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828517504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627457.826072:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627457.826076:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627457.826079:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627457.826084:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11792 00000100:00100000:2.0:1778627457.826088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517632:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627457.826089:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627457.826098:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627457.826126:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627457.826130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627457.826144:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828518016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627457.826148:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627457/real 1778627457] req@ffff954350194700 x1865013828517632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627457.826179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828517632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627457.826186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627457.826189:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627457.826192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627457.826198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627457.826212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627457.826219:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627457/real 1778627457] req@ffff954350197480 x1865013828517760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627457.826226:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828517760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627457.826232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627457.826235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627457.826237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627457.826247:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627457.826262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627457.826269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627457/real 1778627457] req@ffff954350196d80 x1865013828517888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627457.826276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828517888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627457.826281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627457.826284:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627457.826287:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828517888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627457.826309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627457.826320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627457.826327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627457/real 1778627457] req@ffff954350197800 x1865013828518016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627457.826334:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828518016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627457.826340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627457.826342:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627457.826344:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627458.017747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627458.017814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627458.785764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627459.041773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627459.041820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627459.809810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627460.065781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627460.065831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627460.833755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627461.089783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627461.089850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627461.857790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627462.113773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627462.113837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627462.881753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627462.945786:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627462.945794:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627462.945798:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627462.945806:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11797 00000100:00080000:2.0:1778627462.945817:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627462.945822:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627462.945853:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828518144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627462.945878:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627462.945882:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627462.945883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627462.945886:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11797 00000100:00080000:2.0:1778627462.945888:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627462.945891:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627462.945898:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627462.945906:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828518272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627462.945914:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627462.945916:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627462.945932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627462.945935:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11797 00000100:00080000:2.0:1778627462.945939:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627462.945941:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828518144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627462.945944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627462.945949:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627462.945957:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828518400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627462.945965:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627462.945967:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627462.945968:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627462.945971:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11797 00000100:00080000:2.0:1778627462.945973:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627462.945976:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627462.945979:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627462.945986:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828518528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627462.945992:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627462.945994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627462.945996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627462.945998:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11797 00000100:00080000:2.0:1778627462.946001:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627462.946003:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627462.946006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627462.946009:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627462.946022:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828518656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627462.946051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627462/real 1778627462] req@ffff954358788a80 x1865013828518144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627462.946072:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828518144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627462.946083:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627462.946087:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627462.946091:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828518144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627462.946099:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627462.946124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627462.946138:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627462/real 1778627462] req@ffff954358789180 x1865013828518272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627462.946152:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828518272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627462.946161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627462.946165:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627462.946168:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627462.946201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627462.946232:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627462.946244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627462/real 1778627462] req@ffff954358788e00 x1865013828518400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627462.946254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828518400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627462.946260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627462.946262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627462.946264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627462.946268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627462.946282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627462.946288:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627462/real 1778627462] req@ffff954358789500 x1865013828518528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627462.946295:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828518528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627462.946300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627462.946302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627462.946304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627462.946307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627462.946316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627462.946321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627462/real 1778627462] req@ffff954358789c00 x1865013828518656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627462.946338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828518656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627462.946344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627462.946346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627462.946348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627463.137784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627463.137870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627463.905845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627464.161822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627464.161881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627464.929771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627465.185761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627465.185843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627465.953771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627466.209809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627466.209875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627466.977783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627467.233765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627467.233828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627468.001744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627468.065776:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627468.065784:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627468.065788:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627468.065796:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11802 00000100:00080000:2.0:1778627468.065806:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627468.065812:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627468.065845:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828518784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627468.065869:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627468.065872:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627468.065873:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627468.065876:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11802 00000100:00080000:2.0:1778627468.065878:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627468.065880:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627468.065887:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627468.065909:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828518912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627468.065918:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627468.065921:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627468.065923:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627468.065925:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828518784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627468.065927:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11802 00000100:00080000:2.0:1778627468.065933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627468.065936:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627468.065940:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627468.065950:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828519040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627468.065956:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627468.065959:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627468.065960:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627468.065963:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11802 00000100:00080000:2.0:1778627468.065965:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627468.065967:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627468.065970:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627468.065976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828519168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627468.065983:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778627468.065988:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627468.065988:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627468.065992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627468.066003:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11802 00000100:00080000:2.0:1778627468.066006:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627468.066009:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627468.066014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627468.066022:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627468/real 1778627468] req@ffff954358789880 x1865013828518784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627468.066032:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828519296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627468.066039:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828518784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627468.066054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627468.066059:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627468.066065:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828518784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627468.066093:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627468.066115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627468.066124:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627468/real 1778627468] req@ffff954358788700 x1865013828518912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627468.066154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828518912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627468.066160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627468.066162:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627468.066165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828518912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627468.066169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627468.066183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627468.066190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627468/real 1778627468] req@ffff95435878b100 x1865013828519040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627468.066198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828519040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627468.066203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627468.066205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627468.066206:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627468.066210:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627468.066221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627468.066226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627468/real 1778627468] req@ffff95435878bb80 x1865013828519168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627468.066233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828519168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627468.066246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627468.066249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627468.066251:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627468.066255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627468.066267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627468.066273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627468/real 1778627468] req@ffff95435878b480 x1865013828519296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627468.066280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828519296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627468.066285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627468.066287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627468.066289:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627468.257778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627468.257813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627469.025755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778627469.282469:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778627469.282491:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627470.049762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627470.305765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627470.305897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627471.073758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627471.329824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627471.329987:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627472.097758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627472.353997:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778627472.354255:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627473.121762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627473.185775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627473.185785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627473.185790:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627473.185817:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11807 00000100:00080000:3.0:1778627473.185829:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627473.185836:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627473.185875:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828519424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627473.185906:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627473.185914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627473.185916:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627473.185921:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11807 00000100:00080000:3.0:1778627473.185925:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627473.185929:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627473.185938:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627473.185951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828519552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627473.185954:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828519424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627473.185962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627473.185965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627473.185967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627473.185969:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11807 00000100:00080000:3.0:1778627473.185972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627473.185974:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627473.185977:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627473.185985:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828519680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627473.186001:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627473.186003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627473.186005:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627473.186008:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11807 00000100:00080000:3.0:1778627473.186019:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627473.186024:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627473.186029:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778627473.186031:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627473.186043:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828519808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627473.186050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627473.186052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627473.186053:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627473.186057:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11807 00000100:00080000:3.0:1778627473.186060:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627473.186063:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627473.186064:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627473/real 1778627473] req@ffff95434701f480 x1865013828519424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778627473.186067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627473.186078:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828519936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627473.186101:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828519424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627473.186110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627473.186113:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627473.186116:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828519424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627473.186134:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627473.186150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627473.186158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627473/real 1778627473] req@ffff95434701e680 x1865013828519552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627473.186166:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828519552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627473.186172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627473.186174:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627473.186176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627473.186180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627473.186189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627473.186195:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627473/real 1778627473] req@ffff95434701ca80 x1865013828519680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627473.186202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828519680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627473.186216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627473.186218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627473.186220:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627473.186224:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627473.186237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627473.186243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627473/real 1778627473] req@ffff95434701df80 x1865013828519808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627473.186250:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828519808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627473.186256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627473.186257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627473.186259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627473.186262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627473.186272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627473.186277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627473/real 1778627473] req@ffff95434701d180 x1865013828519936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627473.186284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828519936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627473.186289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627473.186291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627473.186297:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828519936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627473.378386:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627473.378446:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627474.145765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627474.401786:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627474.401934:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627475.169797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627475.425758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627475.425824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627476.193767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627476.449753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627476.449785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627477.217782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627477.473755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627477.473780:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627478.241790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627478.305853:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627478.305861:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627478.305865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627478.305873:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11812 00000100:00080000:2.0:1778627478.305883:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627478.305889:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627478.305919:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828520064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627478.305941:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627478.305944:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627478.305959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627478.305963:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11812 00000100:00080000:2.0:1778627478.305968:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627478.305971:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627478.305979:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627478.305991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828520192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627478.305992:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828520064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627478.306005:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627478.306008:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627478.306009:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627478.306012:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11812 00000100:00080000:2.0:1778627478.306014:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627478.306016:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627478.306019:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627478.306029:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828520320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627478.306035:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627478.306038:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627478.306039:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627478.306042:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11812 00000100:00080000:2.0:1778627478.306044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627478.306046:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627478.306050:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627478.306053:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627478.306067:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828520448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627478.306073:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627478.306076:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627478.306077:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627478.306080:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11812 00000100:00080000:2.0:1778627478.306083:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627478.306086:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627478.306089:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627478/real 1778627478] req@ffff95435878aa00 x1865013828520064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627478.306091:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627478.306103:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828520576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627478.306110:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828520064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627478.306119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627478.306122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627478.306126:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828520064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627478.306146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627478.306160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627478.306168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627478/real 1778627478] req@ffff954358788380 x1865013828520192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627478.306175:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828520192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627478.306189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627478.306192:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627478.306194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627478.306198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627478.306211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627478.306217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627478/real 1778627478] req@ffff95435878b480 x1865013828520320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627478.306227:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828520320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627478.306235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627478.306238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627478.306241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627478.306247:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627478.306266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627478.306277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627478/real 1778627478] req@ffff95435878bb80 x1865013828520448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627478.306289:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828520448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627478.306295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627478.306297:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627478.306304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627478.306309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627478.306324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627478.306331:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627478/real 1778627478] req@ffff95435878b100 x1865013828520576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627478.306338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828520576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627478.306343:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627478.306345:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627478.306347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627478.497758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627478.497798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627479.265801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627479.521792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627479.521892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627480.289795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627480.545779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627480.545846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627481.313779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627481.569771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627481.569835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627482.337757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627482.593743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627482.593794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627483.361777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627483.425756:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627483.425763:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627483.425767:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627483.425774:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11817 00000100:00080000:1.0:1778627483.425783:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627483.425789:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627483.425826:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828520704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627483.425847:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627483.425850:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627483.425851:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627483.425855:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11817 00000100:00080000:1.0:1778627483.425857:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627483.425860:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627483.425867:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627483.425877:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828520832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627483.425884:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627483.425887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627483.425890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627483.425892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828520704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627483.425894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11817 00000100:00080000:1.0:1778627483.425899:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627483.425902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627483.425905:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627483.425912:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828520960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627483.425918:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627483.425921:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627483.425922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627483.425930:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11817 00000100:00080000:1.0:1778627483.425943:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627483.425945:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627483.425950:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627483.425953:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627483.425963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828521088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627483.425970:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627483.425973:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627483.425974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627483.425978:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11817 00000100:00080000:1.0:1778627483.425981:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627483.425985:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627483.425986:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627483/real 1778627483] req@ffff954350197100 x1865013828520704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627483.425988:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627483.425999:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828521216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627483.426006:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828520704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627483.426015:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627483.426019:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627483.426029:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828520704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627483.426048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627483.426074:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627483.426083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627483/real 1778627483] req@ffff954350197800 x1865013828520832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627483.426091:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828520832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627483.426097:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627483.426100:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627483.426102:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627483.426106:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627483.426116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627483.426122:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627483/real 1778627483] req@ffff954350196d80 x1865013828520960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627483.426129:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828520960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627483.426134:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627483.426136:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627483.426137:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828520960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627483.426141:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627483.426151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627483.426163:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627483/real 1778627483] req@ffff954350197480 x1865013828521088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627483.426171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828521088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627483.426176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627483.426178:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627483.426186:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627483.426190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627483.426202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627483.426208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627483/real 1778627483] req@ffff954350194700 x1865013828521216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627483.426215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828521216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627483.426221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627483.426223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627483.426225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627483.617756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627483.617826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627484.385752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627484.641745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627484.641809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627485.409748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627485.665771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627485.665852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627486.433771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627486.689743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627486.689808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627487.457764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627487.713770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627487.713828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627488.481762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627488.545743:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627488.545751:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627488.545755:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627488.545763:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11823 00000100:00080000:1.0:1778627488.545774:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627488.545781:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627488.545816:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828521344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627488.545847:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627488.545861:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627488.545863:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627488.545867:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11823 00000100:00080000:1.0:1778627488.545871:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627488.545875:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627488.545883:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627488.545892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828521344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627488.545902:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828521472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627488.545912:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627488.545915:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627488.545917:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627488.545920:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11823 00000100:00080000:1.0:1778627488.545922:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627488.545924:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627488.545928:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627488.545935:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828521600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627488.545942:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627488.545945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627488.545946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627488.545948:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11823 00000100:00080000:1.0:1778627488.545951:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778627488.545954:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627488.545959:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627488.545971:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627488.545982:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828521728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627488.545990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627488.545992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627488.545993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627488.545997:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627488/real 1778627488] req@ffff954350194000 x1865013828521344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627488.546000:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11823 00000100:00080000:1.0:1778627488.546005:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627488.546011:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627488.546014:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627488.546019:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828521344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627488.546023:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828521856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627488.546039:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627488.546045:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627488.546049:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828521344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627488.546068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627488.546085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627488.546093:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627488/real 1778627488] req@ffff954350194a80 x1865013828521472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627488.546101:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828521472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627488.546107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627488.546109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627488.546111:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627488.546115:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627488.546124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627488.546130:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627488/real 1778627488] req@ffff954350194380 x1865013828521600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627488.546137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828521600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627488.546153:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627488.546168:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627488.546171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627488.546176:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627488.546192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627488.546199:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627488/real 1778627488] req@ffff954350194e00 x1865013828521728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627488.546207:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828521728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627488.546212:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627488.546215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627488.546216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627488.546220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627488.546229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627488.546234:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627488/real 1778627488] req@ffff954350195500 x1865013828521856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627488.546246:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828521856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627488.546254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627488.546258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627488.546267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828521856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627488.737752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627488.737794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627489.505788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627489.761755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627489.761815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627490.529745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627490.785741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627490.785819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627491.553771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627491.809755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627491.809839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627492.577811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627492.833773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627492.833838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627493.601754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627493.665781:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627493.665788:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627493.665793:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627493.665801:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11828 00000100:00080000:1.0:1778627493.665810:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627493.665816:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627493.665848:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828521984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627493.665871:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627493.665874:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627493.665876:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627493.665878:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11828 00000100:00080000:1.0:1778627493.665895:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627493.665898:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627493.665905:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627493.665914:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828522112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627493.665921:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627493.665923:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627493.665925:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627493.665927:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11828 00000100:00080000:1.0:1778627493.665930:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627493.665932:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627493.665935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627493.665941:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828522240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627493.665948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627493.665951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627493.665952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627493.665956:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11828 00000100:00080000:1.0:1778627493.665960:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627493.665961:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828521984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627493.665965:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627493.665973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627493.665981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828522368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627493.666008:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627493.666011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627493.666013:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627493.666017:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11828 00000100:00080000:1.0:1778627493.666034:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627493.666039:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627493.666044:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627493.666048:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627493.666081:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828522496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627493.666097:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627493/real 1778627493] req@ffff954350195c00 x1865013828521984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627493.666126:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828521984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627493.666135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627493.666138:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627493.666141:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828521984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627493.666147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627493.666169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627493.666189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627493/real 1778627493] req@ffff954350195880 x1865013828522112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627493.666198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828522112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627493.666204:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627493.666206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627493.666216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627493.666240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627493.666257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627493.666265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627493/real 1778627493] req@ffff954350197b80 x1865013828522240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627493.666273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828522240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627493.666278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627493.666280:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627493.666282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627493.666286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627493.666296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627493.666301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627493/real 1778627493] req@ffff954350195180 x1865013828522368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627493.666308:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828522368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627493.666313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627493.666315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627493.666316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627493.666320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627493.666328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627493.666341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627493/real 1778627493] req@ffff954350196a00 x1865013828522496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627493.666349:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828522496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627493.666354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627493.666356:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627493.666358:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627493.857739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627493.857840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627494.625778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627494.881752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627494.881812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627495.649750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627495.905742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627495.905805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627496.673757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627496.929751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627496.929795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627497.697781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627497.953771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627497.953826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627498.721807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627498.785737:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627498.785744:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627498.785748:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627498.785757:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11833 00000100:00080000:1.0:1778627498.785766:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627498.785805:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627498.785837:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828522624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627498.785863:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627498.785867:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627498.785869:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627498.785874:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11833 00000100:00080000:1.0:1778627498.785878:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627498.785882:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627498.785914:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627498.785917:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828522624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627498.785932:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828522752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627498.785942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627498.785944:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627498.785945:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627498.785948:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11833 00000100:00080000:1.0:1778627498.785963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627498.785967:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627498.785971:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627498.785980:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828522880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627498.785987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627498.785989:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627498.785990:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627498.785995:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627498.785995:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11833 00000100:00080000:1.0:1778627498.786001:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627498.786004:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627498.786009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627498.786017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828523008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627498.786024:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627498.786027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627498.786031:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627498/real 1778627498] req@ffff954350195f80 x1865013828522624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627498.786035:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627498.786045:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11833 00000100:00080000:1.0:1778627498.786051:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627498.786054:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828522624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627498.786057:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627498.786068:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627498.786070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627498.786077:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627498.786086:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828523136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627498.786093:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828522624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627498.786126:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627498.786151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627498.786168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627498/real 1778627498] req@ffff954350196a00 x1865013828522752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627498.786177:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828522752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627498.786183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627498.786185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627498.786187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627498.786191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627498.786205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627498.786214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627498/real 1778627498] req@ffff954350195180 x1865013828522880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627498.786226:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828522880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627498.786232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627498.786235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627498.786238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828522880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627498.786244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627498.786262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627498.786273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627498/real 1778627498] req@ffff954350197b80 x1865013828523008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627498.786284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828523008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627498.786298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627498.786301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627498.786304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627498.786311:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627498.786331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627498.786341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627498/real 1778627498] req@ffff954350195880 x1865013828523136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627498.786352:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828523136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627498.786360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627498.786363:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627498.786366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627498.977760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627498.977798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627499.745762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627500.001761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627500.001821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627500.769820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627501.025759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627501.025831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627501.793740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627502.049744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627502.049792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627502.817819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627503.073753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627503.073815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627503.841732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627503.905805:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627503.905813:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627503.905817:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627503.905825:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11838 00000100:00080000:2.0:1778627503.905835:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627503.905841:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627503.905891:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828523264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627503.905941:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627503.905945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627503.905947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627503.905952:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11838 00000100:00080000:2.0:1778627503.905956:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627503.905977:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627503.905986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627503.906000:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828523392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627503.906024:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828523264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627503.906038:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627503.906042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627503.906044:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627503.906049:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11838 00000100:00080000:2.0:1778627503.906053:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627503.906057:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627503.906063:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627503.906075:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828523520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627503.906084:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627503.906088:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627503.906090:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627503.906093:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11838 00000100:00080000:2.0:1778627503.906098:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627503.906103:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627503.906110:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627503.906127:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828523648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627503.906140:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627503.906144:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627503.906178:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627503.906184:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11838 00000100:00080000:2.0:1778627503.906198:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627503.906204:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627503.906229:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627503.906242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627503.906249:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828523776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627503.906281:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627503.906297:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627503.906313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627504.097809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627504.097947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627504.865750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627505.121798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627505.121888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627505.889748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627506.145748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627506.146010:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627506.913770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778627506.978115:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778627506.978174:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778627506.978206:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627506.978250:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627506.978262:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627506.978301:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627506.978316:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627506.978341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627503/real 1778627506] req@ffff954358788e00 x1865013828523776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627506.978380:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828523776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627506.978392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627506.978397:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627506.978401:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627506.978410:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627503/real 1778627506] req@ffff954358789880 x1865013828523392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627506.978427:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828523392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627506.978440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627506.978445:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627506.978449:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627506.978459:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627503/real 1778627506] req@ffff954358789c00 x1865013828523520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627506.978476:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828523520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627506.978502:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627506.978507:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627506.978512:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627506.978541:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627503/real 1778627506] req@ffff954358789500 x1865013828523648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627506.978553:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828523648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627506.978560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627506.978563:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627506.978567:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828523648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627506.978575:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627503/real 1778627506] req@ffff954358788700 x1865013828523264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627506.978597:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828523264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627506.978608:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627506.978613:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627506.978618:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828523264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627507.169811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627507.169978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627507.937753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627508.193744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627508.193831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627508.961777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627509.025815:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627509.025826:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627509.025830:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627509.025838:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11843 00000100:00080000:2.0:1778627509.025848:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627509.025854:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627509.025895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828523904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627509.025917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627509.025920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627509.025921:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627509.025937:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11843 00000100:00080000:2.0:1778627509.025941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627509.025945:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627509.025953:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627509.025965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828524032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627509.025966:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828523904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627509.025980:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627509.025982:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627509.025984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627509.025988:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11843 00000100:00080000:2.0:1778627509.025990:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627509.025992:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627509.025996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627509.026005:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828524160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627509.026012:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627509.026014:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627509.026016:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627509.026018:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11843 00000100:00080000:2.0:1778627509.026020:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627509.026025:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627509.026026:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778627509.026029:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627509.026038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828524288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627509.026051:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627509.026053:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627509.026055:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627509.026058:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11843 00000100:00080000:2.0:1778627509.026062:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627509.026065:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627509.026070:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627509/real 1778627509] req@ffff954358789180 x1865013828523904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627509.026071:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627509.026081:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828524416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627509.026116:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828523904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627509.026128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627509.026140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627509.026145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828523904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627509.026169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627509.026192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627509.026206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627509/real 1778627509] req@ffff954358788a80 x1865013828524032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627509.026220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828524032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627509.026229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627509.026233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627509.026237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627509.026244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627509.026266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627509.026278:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627509/real 1778627509] req@ffff95435878b800 x1865013828524160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627509.026290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828524160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627509.026305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627509.026308:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627509.026312:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627509.026318:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627509.026348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627509.026359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627509/real 1778627509] req@ffff95435878a300 x1865013828524288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627509.026373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828524288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627509.026380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627509.026383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627509.026387:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627509.026394:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627509.026414:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627509.026426:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627509/real 1778627509] req@ffff95435878ad80 x1865013828524416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627509.026439:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828524416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627509.026448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627509.026452:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627509.026455:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627509.217744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627509.217837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627509.985771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627510.241748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627510.241831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627511.009764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627511.265782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627511.265858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627512.033811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627512.289806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627512.289928:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627513.057741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627513.313770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627513.313849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627514.081784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627514.145753:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627514.145764:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627514.145769:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627514.145780:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11848 00000100:00080000:1.0:1778627514.145792:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627514.145799:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627514.145838:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828524544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627514.145868:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627514.145872:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627514.145888:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627514.145892:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11848 00000100:00080000:1.0:1778627514.145897:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627514.145900:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627514.145910:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627514.145916:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828524544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627514.145927:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828524672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627514.145941:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627514.145945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627514.145948:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627514.145952:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11848 00000100:00080000:1.0:1778627514.145959:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627514.145964:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627514.145969:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627514.145973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627514.145991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828524800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627514.146001:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627514/real 1778627514] req@ffff954350195c00 x1865013828524544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627514.146004:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627514.146015:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627514.146017:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627514.146025:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11848 00000100:00080000:1.0:1778627514.146033:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627514.146037:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627514.146042:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627514.146071:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828524928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627514.146078:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627514.146081:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627514.146083:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627514.146086:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11848 00000100:00100000:0.0:1778627514.146090:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828524544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627514.146092:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627514.146100:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778627514.146103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778627514.146105:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627514.146109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627514.146114:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828524544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627514.146119:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828525056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627514.146135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627514.146155:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627514.146164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627514/real 1778627514] req@ffff954350195500 x1865013828524672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627514.146172:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828524672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627514.146178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627514.146180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627514.146182:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627514.146186:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627514.146196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627514.146201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627514/real 1778627514] req@ffff954350194e00 x1865013828524800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627514.146207:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828524800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627514.146222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627514.146224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627514.146226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627514.146230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627514.146244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627514.146250:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627514/real 1778627514] req@ffff954350194380 x1865013828524928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627514.146257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828524928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627514.146262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627514.146264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627514.146265:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828524928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627514.146269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627514.146278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627514.146283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627514/real 1778627514] req@ffff954350194a80 x1865013828525056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627514.146299:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828525056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627514.146305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627514.146307:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627514.146314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627514.337758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627514.337803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627515.105740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627515.361773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627515.361821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627516.129766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627516.385731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627516.385781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627517.153763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627517.409742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627517.409805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627518.177764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627518.433759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627518.433816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627519.201734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627519.265784:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627519.265792:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627519.265796:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627519.265804:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11853 00000100:00080000:1.0:1778627519.265814:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627519.265820:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627519.265859:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828525184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627519.265877:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627519.265880:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627519.265881:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627519.265884:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11853 00000100:00080000:1.0:1778627519.265886:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627519.265902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627519.265910:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627519.265921:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828525312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627519.265922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828525184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627519.265936:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627519.265938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627519.265939:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627519.265942:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11853 00000100:00080000:1.0:1778627519.265946:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627519.265948:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627519.265951:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627519.265960:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828525440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627519.265966:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627519.265968:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627519.265970:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627519.265972:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11853 00000100:00080000:1.0:1778627519.265975:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627519.265980:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627519.265983:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627519.265987:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627519.266000:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828525568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627519.266011:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627519.266014:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627519/real 1778627519] req@ffff954350194000 x1865013828525184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627519.266026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627519.266030:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627519.266036:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11853 00000100:00100000:0.0:1778627519.266037:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828525184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627519.266043:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627519.266051:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627519.266051:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627519.266057:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627519.266058:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627519.266064:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828525184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627519.266073:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828525696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627519.266089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627519.266126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627519.266139:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627519/real 1778627519] req@ffff954350194700 x1865013828525312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627519.266149:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828525312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627519.266155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627519.266157:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627519.266160:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627519.266164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627519.266178:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627519.266191:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627519/real 1778627519] req@ffff954350197480 x1865013828525440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627519.266199:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828525440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627519.266205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627519.266207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627519.266209:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627519.266213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627519.266225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627519.266230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627519/real 1778627519] req@ffff954350196d80 x1865013828525568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627519.266237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828525568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627519.266242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627519.266243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627519.266245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627519.266248:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627519.266257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627519.266262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627519/real 1778627519] req@ffff954350197800 x1865013828525696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627519.266269:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828525696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627519.266279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627519.266281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627519.266283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627519.457770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627519.457819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627520.225766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627520.481780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627520.481848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627521.249758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627521.505765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627521.505825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627522.273772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627522.529756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627522.529821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627523.297730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627523.553746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627523.553804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627524.321778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627524.385765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627524.385773:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627524.385777:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627524.385785:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11858 00000100:00080000:3.0:1778627524.385794:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627524.385800:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627524.385830:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828525824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627524.385854:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627524.385858:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627524.385860:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627524.385864:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11858 00000100:00080000:3.0:1778627524.385868:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627524.385872:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627524.385882:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627524.385897:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828525824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627524.385898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828525952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627524.385916:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627524.385919:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627524.385934:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627524.385937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11858 00000100:00080000:3.0:1778627524.385941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627524.385944:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627524.385948:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627524.385958:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828526080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627524.385965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778627524.385969:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627524.385978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627524.385980:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627524.385984:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11858 00000100:00080000:3.0:1778627524.385987:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627524.385990:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627524.385994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627524.386004:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627524/real 1778627524] req@ffff95434701d880 x1865013828525824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627524.386005:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828526208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627524.386032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627524.386034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627524.386036:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627524.386040:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828525824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627524.386042:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11858 00000100:00080000:3.0:1778627524.386047:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627524.386051:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627524.386058:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627524.386064:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627524.386071:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:3.0:1778627524.386074:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828526336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627524.386082:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828525824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627524.386107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627524.386125:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627524.386134:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627524/real 1778627524] req@ffff95434701ea00 x1865013828525952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627524.386142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828525952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627524.386148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627524.386151:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627524.386153:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828525952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627524.386166:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627524.386179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627524.386185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627524/real 1778627524] req@ffff95434701c000 x1865013828526080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627524.386193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828526080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627524.386198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627524.386200:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627524.386202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627524.386205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627524.386215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627524.386220:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627524/real 1778627524] req@ffff95434701c700 x1865013828526208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627524.386227:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828526208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627524.386232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627524.386234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627524.386235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627524.386238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627524.386247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627524.386251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627524/real 1778627524] req@ffff95434701f800 x1865013828526336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627524.386264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828526336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627524.386269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627524.386271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627524.386273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627524.577763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627524.577812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627525.345728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627525.601746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627525.601796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627526.369760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627526.625773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627526.625844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627527.393752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627527.649740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627527.649789:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627528.417744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627528.673776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627528.673845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627529.441740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627529.505767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627529.505774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627529.505778:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627529.505786:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11864 00000100:00080000:3.0:1778627529.505796:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627529.505801:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627529.505833:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828526464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627529.505857:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627529.505859:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627529.505861:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627529.505864:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11864 00000100:00080000:3.0:1778627529.505866:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627529.505868:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627529.505875:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627529.505899:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828526592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627529.505906:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627529.505923:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627529.505925:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627529.505928:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11864 00000100:00080000:3.0:1778627529.505932:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627529.505935:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627529.505939:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627529.505965:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828526464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627529.505965:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828526720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627529.505979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627529.505981:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627529.505982:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627529.505986:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11864 00000100:00080000:3.0:1778627529.505988:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627529.505991:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627529.505994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627529.506004:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828526848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627529.506009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627529.506011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627529.506012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627529.506016:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11864 00000100:00080000:3.0:1778627529.506018:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627529.506020:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627529.506028:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778627529.506031:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627529.506042:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828526976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627529.506064:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627529/real 1778627529] req@ffff95434701d500 x1865013828526464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627529.506078:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828526464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627529.506086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627529.506089:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627529.506092:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828526464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627529.506097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627529.506110:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627529.506117:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627529/real 1778627529] req@ffff95434701c380 x1865013828526592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627529.506124:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828526592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627529.506141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627529.506143:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627529.506145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627529.506163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627529.506176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627529.506182:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627529/real 1778627529] req@ffff95434701f100 x1865013828526720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627529.506189:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828526720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627529.506194:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627529.506196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627529.506215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627529.506219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627529.506230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627529.506236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627529/real 1778627529] req@ffff95434701f800 x1865013828526848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627529.506244:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828526848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627529.506249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627529.506251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627529.506259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627529.506280:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627529.506297:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627529.506305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627529/real 1778627529] req@ffff95434701c700 x1865013828526976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627529.506315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828526976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627529.506322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627529.506325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627529.506327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828526976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627529.697728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627529.697801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627530.465777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627530.721785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627530.721848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627531.489741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627531.745747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627531.745806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627532.513767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627532.769773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627532.769843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627533.537799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627533.793754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627533.793798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627534.561753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627534.625776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627534.625787:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627534.625790:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627534.625798:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11869 00000100:00080000:0.0:1778627534.625809:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627534.625815:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627534.625846:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828527104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627534.625872:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627534.625874:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627534.625876:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627534.625879:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11869 00000100:00080000:0.0:1778627534.625881:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627534.625897:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627534.625906:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627534.625910:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828527104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627534.625918:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828527232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627534.625930:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627534.625933:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627534.625934:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627534.625937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11869 00000100:00080000:0.0:1778627534.625941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627534.625943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627534.625946:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627534.625954:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828527360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627534.625960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627534.625963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627534.625965:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627534.625967:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11869 00000100:00080000:0.0:1778627534.625970:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627534.625972:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627534.625975:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627534.625976:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627534.625986:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828527488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627534.625994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627534.626001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627534.626003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627534.626007:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11869 00000100:00080000:0.0:1778627534.626010:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627534.626013:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627534/real 1778627534] req@ffff954342f49880 x1865013828527104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627534.626016:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627534.626024:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627534.626034:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828527616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627534.626037:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828527104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627534.626063:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627534.626068:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627534.626071:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828527104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627534.626092:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627534.626110:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627534.626122:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627534/real 1778627534] req@ffff954342f4bb80 x1865013828527232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627534.626134:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828527232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627534.626144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627534.626147:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627534.626151:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627534.626158:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627534.626177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627534.626185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627534/real 1778627534] req@ffff954342f49180 x1865013828527360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627534.626203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828527360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627534.626209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627534.626211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627534.626213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627534.626217:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627534.626239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627534.626264:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627534/real 1778627534] req@ffff954342f48000 x1865013828527488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627534.626274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828527488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627534.626279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627534.626281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627534.626283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627534.626287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627534.626300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627534.626306:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627534/real 1778627534] req@ffff954342f48380 x1865013828527616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627534.626313:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828527616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627534.626327:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627534.626330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627534.626332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627534.817756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627534.817815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627535.585748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627535.842040:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627535.842548:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627536.609739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627536.866134:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627536.866272:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627537.633754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627537.889843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627537.889949:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627538.657767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627538.913771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627538.913832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627539.681747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627539.745767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627539.745775:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627539.745779:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627539.745787:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11874 00000100:00080000:2.0:1778627539.745798:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627539.745803:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627539.745835:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828527744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627539.745859:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627539.745861:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627539.745863:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627539.745866:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11874 00000100:00080000:2.0:1778627539.745868:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627539.745884:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627539.745892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828527744:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778627539.745892:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627539.745905:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828527872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627539.745913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627539.745916:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627539.745917:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627539.745920:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11874 00000100:00080000:2.0:1778627539.745923:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627539.745925:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627539.745929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627539.745936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828528000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627539.745943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627539.745945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627539.745946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627539.745951:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627539.745953:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11874 00000100:00080000:2.0:1778627539.745956:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627539.745958:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627539.745962:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627539.745970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828528128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627539.745977:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627539.745984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627539.745985:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627539/real 1778627539] req@ffff954358789f80 x1865013828527744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627539.745987:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627539.745994:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11874 00000100:00080000:2.0:1778627539.746000:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627539.746004:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627539.746008:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627539.746009:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828527744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627539.746018:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828528256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627539.746019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627539.746027:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627539.746034:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828527744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627539.746053:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627539.746068:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627539.746076:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627539/real 1778627539] req@ffff95435878a680 x1865013828527872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627539.746096:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828527872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627539.746103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627539.746105:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627539.746107:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828527872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627539.746111:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627539.746123:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627539.746129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627539/real 1778627539] req@ffff95435878ad80 x1865013828528000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627539.746136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828528000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627539.746141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627539.746160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627539.746163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627539.746167:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627539.746182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627539.746189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627539/real 1778627539] req@ffff95435878a300 x1865013828528128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627539.746197:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828528128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627539.746211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627539.746213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627539.746216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627539.746220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627539.746233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627539.746246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627539/real 1778627539] req@ffff95435878b800 x1865013828528256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627539.746254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828528256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627539.746259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627539.746261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627539.746263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627539.937890:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627539.937959:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627540.705763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627540.961768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627540.961807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627541.729814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627541.985770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627541.985794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627542.753740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627543.009753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627543.009792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627543.777775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627544.033786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627544.033816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627544.801760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627544.865756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627544.865766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627544.865769:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627544.865777:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11879 00000100:00080000:2.0:1778627544.865788:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627544.865794:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627544.865829:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828528384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627544.865882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627544.865886:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627544.865889:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627544.865893:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11879 00000100:00080000:2.0:1778627544.865897:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627544.865901:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627544.865909:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828528384:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778627544.865911:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627544.865929:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828528512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627544.865941:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627544.865945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627544.865947:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627544.865951:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11879 00000100:00080000:2.0:1778627544.865955:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627544.865959:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627544.865964:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778627544.865973:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627544.865975:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828528640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627544.865982:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627544.865985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627544.865986:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627544.865997:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11879 00000100:00080000:2.0:1778627544.866005:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627544.866009:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627544.866014:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627544/real 1778627544] req@ffff954358788a80 x1865013828528384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627544.866014:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627544.866027:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828528768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627544.866034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627544.866037:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627544.866039:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627544.866041:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828528384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627544.866054:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11879 00000100:00080000:2.0:1778627544.866057:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627544.866059:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627544.866062:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627544.866074:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828528896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627544.866077:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627544.866087:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627544.866093:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828528384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627544.866117:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627544.866142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627544.866155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627544/real 1778627544] req@ffff954358789180 x1865013828528512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627544.866169:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828528512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627544.866179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627544.866183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627544.866186:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627544.866193:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627544.866210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627544.866217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627544/real 1778627544] req@ffff954358788700 x1865013828528640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627544.866225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828528640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627544.866230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627544.866233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627544.866235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627544.866246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627544.866260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627544.866266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627544/real 1778627544] req@ffff954358789500 x1865013828528768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627544.866274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828528768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627544.866279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627544.866281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627544.866283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627544.866286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627544.866295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627544.866300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627544/real 1778627544] req@ffff954358789c00 x1865013828528896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627544.866307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828528896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627544.866312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627544.866314:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627544.866316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828528896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627545.057744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627545.057766:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627545.825798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627546.081783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627546.081819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627546.849751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627547.105742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627547.105794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627547.873746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627548.129768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627548.129807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627548.897744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627549.153752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627549.153817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627549.921779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627549.985784:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627549.985792:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627549.985796:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627549.985803:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11884 00000100:00080000:1.0:1778627549.985812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627549.985818:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627549.985849:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828529024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627549.985877:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627549.985880:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627549.985881:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627549.985884:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11884 00000100:00080000:1.0:1778627549.985887:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627549.985889:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627549.985897:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627549.985916:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828529024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627549.985923:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828529152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627549.985938:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627549.985953:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627549.985955:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627549.985959:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11884 00000100:00080000:1.0:1778627549.985963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627549.985967:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627549.985971:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627549.985981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828529280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627549.985988:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627549.985991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627549.985995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627549.985996:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627549.986000:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11884 00000100:00080000:1.0:1778627549.986004:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627549.986007:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627549.986010:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627549.986019:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828529408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627549.986026:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627549.986030:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627549.986031:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627549/real 1778627549] req@ffff954350197100 x1865013828529024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627549.986032:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627549.986046:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11884 00000100:00080000:1.0:1778627549.986050:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627549.986053:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828529024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627549.986057:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627549.986065:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627549.986069:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627549.986074:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627549.986077:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828529536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627549.986081:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828529024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627549.986107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627549.986124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627549.986132:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627549/real 1778627549] req@ffff954350197800 x1865013828529152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627549.986141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828529152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627549.986146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627549.986149:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627549.986151:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627549.986157:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627549.986176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627549.986187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627549/real 1778627549] req@ffff954350196d80 x1865013828529280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627549.986214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828529280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627549.986222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627549.986224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627549.986226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627549.986231:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627549.986250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627549.986274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627549/real 1778627549] req@ffff954350197480 x1865013828529408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627549.986288:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828529408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627549.986296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627549.986300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627549.986303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627549.986309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627549.986327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627549.986335:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627549/real 1778627549] req@ffff954350194700 x1865013828529536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627549.986343:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828529536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627549.986364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627549.986367:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627549.986371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627550.177735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627550.177782:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627550.945734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627551.201769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627551.201842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627551.969737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627552.225770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627552.225835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627552.993725:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627553.249748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627553.249835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627554.017783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627554.273758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627554.273880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627555.041785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627555.105757:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627555.105767:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627555.105772:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627555.105782:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11889 00000100:00080000:2.0:1778627555.105793:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627555.105801:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627555.105835:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828529664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627555.105867:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627555.105870:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627555.105872:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627555.105875:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11889 00000100:00080000:2.0:1778627555.105878:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627555.105882:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627555.105890:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627555.105901:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828529792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627555.105909:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828529664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627555.105926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627555.105928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627555.105930:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627555.105933:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11889 00000100:00080000:2.0:1778627555.105937:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627555.105940:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627555.105944:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627555.105954:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828529920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627555.105960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627555.105964:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1778627555.105965:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627555.105966:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627555.105971:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11889 00000100:00080000:2.0:1778627555.105974:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627555.105977:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627555.105980:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627555.105988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828530048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627555.105994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627555.105997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627555.105998:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627555.106001:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11889 00000100:00100000:0.0:1778627555.106007:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627555/real 1778627555] req@ffff954358789880 x1865013828529664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627555.106009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627555.106015:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627555.106022:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627555.106031:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828530176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627555.106038:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828529664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627555.106050:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627555.106054:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627555.106059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828529664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627555.106083:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627555.106109:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627555.106122:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627555/real 1778627555] req@ffff954358788e00 x1865013828529792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627555.106131:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828529792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627555.106137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627555.106139:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627555.106142:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627555.106153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627555.106171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627555.106182:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627555/real 1778627555] req@ffff95435878b100 x1865013828529920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627555.106195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828529920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627555.106203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627555.106207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627555.106210:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828529920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627555.106217:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627555.106236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627555.106243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627555/real 1778627555] req@ffff95435878bb80 x1865013828530048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627555.106251:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828530048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627555.106265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627555.106268:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627555.106270:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627555.106274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627555.106285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627555.106290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627555/real 1778627555] req@ffff95435878b480 x1865013828530176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627555.106304:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828530176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627555.106310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627555.106312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627555.106314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627555.297736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627555.297800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627556.065772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627556.321785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627556.321852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627557.089819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627557.345762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627557.345855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627558.113752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627558.369774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627558.369840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627559.137742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627559.393751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627559.393812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627560.161731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627560.225748:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627560.225756:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627560.225760:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627560.225767:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11894 00000100:00080000:2.0:1778627560.225777:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627560.225783:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627560.225813:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828530304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627560.225840:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627560.225843:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627560.225845:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627560.225848:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11894 00000100:00080000:2.0:1778627560.225850:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627560.225853:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627560.225869:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627560.225879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828530432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627560.225884:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828530304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627560.225892:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627560.225895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627560.225896:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627560.225899:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11894 00000100:00080000:2.0:1778627560.225902:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627560.225905:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627560.225908:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627560.225916:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828530560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627560.225924:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627560.225926:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627560.225927:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627560.225930:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11894 00000100:00080000:2.0:1778627560.225932:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627560.225934:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627560.225937:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627560.225944:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828530688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627560.225950:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627560.225956:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627560.225958:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627560.225963:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627560.225967:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11894 00000100:00080000:2.0:1778627560.225971:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627560.225974:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627560.225977:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627560.225985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828530816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627560.225998:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627560/real 1778627560] req@ffff954358788380 x1865013828530304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627560.226016:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828530304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627560.226026:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627560.226030:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627560.226035:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828530304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627560.226056:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627560.226077:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627560.226096:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627560/real 1778627560] req@ffff95435878aa00 x1865013828530432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627560.226108:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828530432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627560.226130:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627560.226133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627560.226137:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627560.226144:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627560.226164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627560.226174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627560/real 1778627560] req@ffff95435878b480 x1865013828530560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627560.226184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828530560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627560.226192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627560.226196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627560.226199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627560.226205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627560.226223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627560.226232:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627560/real 1778627560] req@ffff95435878bb80 x1865013828530688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627560.226243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828530688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627560.226251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627560.226254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627560.226263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627560.226269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627560.226287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627560.226296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627560/real 1778627560] req@ffff95435878b100 x1865013828530816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627560.226307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828530816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627560.226316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627560.226319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627560.226321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828530816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627560.417785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627560.417798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627561.185786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627561.441761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627561.441819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627562.209765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627562.465762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627562.465830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627563.233834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627563.489791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627563.489836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627564.257765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627564.513796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627564.513860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627565.281762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627565.345844:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627565.345859:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627565.345865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627565.345895:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11899 00000100:00080000:1.0:1778627565.345910:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627565.345919:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627565.345984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828530944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627565.346042:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627565.346047:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627565.346050:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627565.346077:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11899 00000100:00080000:1.0:1778627565.346084:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627565.346090:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627565.346100:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828530944:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627565.346103:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627565.346147:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828531072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627565.346165:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627565.346170:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627565.346173:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627565.346179:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11899 00000800:00000100:0.0:1778627565.346185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627565.346185:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627565.346197:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627565.346205:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627565.346226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627565/real 1778627565] req@ffff954350194000 x1865013828530944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627565.346226:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828531200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627565.346315:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627565.346317:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828530944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627565.346326:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627565.346336:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627565.346342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627565.346348:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11899 00000100:00080000:0.0:1778627565.346351:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627565.346359:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627565.346360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828530944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627565.346377:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627565.346385:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627565.346392:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627565.346407:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828531328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627565.346443:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627565.346451:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627565.346457:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627565.346461:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627565/real 1778627565] req@ffff954350194a80 x1865013828531072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627565.346461:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627565.346481:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11899 00000100:00100000:0.0:1778627565.346492:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828531072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627565.346494:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627565.346506:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627565.346507:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778627565.346513:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627565.346519:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531072:192.168.204.102@tcp:38:kworker.0 00000008:00100000:1.0:1778627565.346521:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627565.346526:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627565.346549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627565.346549:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828531456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627565.346562:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627565/real 1778627565] req@ffff954350194380 x1865013828531200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627565.346593:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828531200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627565.346600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627565.346603:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627565.346605:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627565.346619:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627565.346634:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627565.346664:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627565/real 1778627565] req@ffff954350194e00 x1865013828531328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627565.346736:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828531328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627565.346746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627565.346748:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627565.346751:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627565.346756:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627565.346781:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627565.346790:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627565/real 1778627565] req@ffff954350195500 x1865013828531456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627565.346798:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828531456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627565.346805:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627565.346807:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627565.346809:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627565.537807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627565.537883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627566.305726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627566.561763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627566.561815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627567.329762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627567.585806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627567.585909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627568.353740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627568.609758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627568.609849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627569.377766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627569.633775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627569.633842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627570.401728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627570.465755:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627570.465763:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627570.465767:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627570.465775:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11904 00000100:00080000:1.0:1778627570.465812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627570.465818:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627570.465854:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828531584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627570.465879:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627570.465882:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627570.465898:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627570.465903:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11904 00000100:00080000:1.0:1778627570.465907:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627570.465927:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627570.465935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627570.465947:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828531584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627570.465947:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828531712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627570.465962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627570.465964:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627570.465965:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627570.465968:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11904 00000100:00080000:1.0:1778627570.465970:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627570.465973:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627570.465976:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627570.465985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828531840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627570.465991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627570.465993:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627570.465995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627570.465998:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11904 00000100:00080000:1.0:1778627570.466000:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627570.466002:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627570.466005:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627570.466017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828531968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627570.466024:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627570.466038:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627570.466040:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627570.466042:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11904 00000100:00080000:1.0:1778627570.466044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627570.466047:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627570.466051:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531712:192.168.204.102@tcp:38:kworker.0 00000008:00100000:1.0:1778627570.466053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627570.466063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828532096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627570.466077:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627570.466105:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627570.466118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627570.657808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627570.657900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627571.425824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627571.681795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627571.681864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627572.449776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627572.705742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627572.705805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627573.473772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627573.537955:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627573.537990:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627573.538007:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627573.538047:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627573.538057:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627573.538065:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627573.538076:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627573.538106:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627570/real 1778627573] req@ffff954350196a00 x1865013828532096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627573.538133:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828532096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627573.538145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627573.538151:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627573.538156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627573.538164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627570/real 1778627573] req@ffff954350195880 x1865013828531712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627573.538177:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828531712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627573.538186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627573.538190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627573.538194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627573.538201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627570/real 1778627573] req@ffff954350197b80 x1865013828531840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627573.538213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828531840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627573.538221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627573.538225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627573.538229:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627573.538255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627570/real 1778627573] req@ffff954350195180 x1865013828531968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627573.538278:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828531968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627573.538287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627573.538291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627573.538294:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828531968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627573.538301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627570/real 1778627573] req@ffff954350195c00 x1865013828531584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627573.538314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828531584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627573.538323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627573.538326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627573.538330:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828531584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627573.729778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627573.729845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627574.497785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627574.753748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627574.753804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627575.521775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627575.585820:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627575.585828:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627575.585832:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627575.585840:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11910 00000100:00080000:2.0:1778627575.585849:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627575.585866:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627575.585895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828532224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627575.585919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627575.585923:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627575.585924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627575.585929:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11910 00000100:00080000:2.0:1778627575.585933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627575.585937:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627575.585945:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627575.585958:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828532352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627575.585970:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627575.585973:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627575.585977:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627575.585978:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828532224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627575.585982:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11910 00000100:00080000:2.0:1778627575.585991:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627575.585995:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627575.586000:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627575.586013:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828532480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627575.586024:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627575.586027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627575.586036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627575.586041:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11910 00000100:00080000:2.0:1778627575.586047:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778627575.586049:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627575.586054:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627575.586061:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627575.586106:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828532608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627575.586113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627575/real 1778627575] req@ffff954358788e00 x1865013828532224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627575.586122:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627575.586130:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627575.586132:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627575.586138:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11910 00000100:00100000:0.0:1778627575.586143:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828532224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627575.586143:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627575.586153:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778627575.586163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1778627575.586163:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627575.586178:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828532736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627575.586181:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627575.586190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828532224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627575.586220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627575.586247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627575.586260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627575/real 1778627575] req@ffff954358789880 x1865013828532352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627575.586272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828532352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627575.586280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627575.586283:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627575.586286:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627575.586292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627575.586311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627575.586320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627575/real 1778627575] req@ffff954358789c00 x1865013828532480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627575.586331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828532480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627575.586338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627575.586340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627575.586344:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627575.586349:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627575.586385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627575.586397:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627575/real 1778627575] req@ffff954358789500 x1865013828532608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627575.586408:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828532608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627575.586416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627575.586420:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627575.586423:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627575.586429:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627575.586468:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627575.586480:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627575/real 1778627575] req@ffff954358788700 x1865013828532736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627575.586491:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828532736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627575.586499:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627575.586503:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627575.586506:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627575.777753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627575.777856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627576.545788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627576.801781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627576.801836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627577.569733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627577.825755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627577.825799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627578.593780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627578.849727:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627578.849820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627579.617750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627579.873761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627579.873820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627580.641790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627580.705809:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627580.705817:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627580.705821:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627580.705828:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11915 00000100:00080000:1.0:1778627580.705838:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627580.705862:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627580.705895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828532864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627580.705926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627580.705930:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627580.705932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627580.705936:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11915 00000100:00080000:1.0:1778627580.705941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627580.705944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627580.705952:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627580.705965:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828532864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627580.705966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828532992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627580.705980:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627580.705982:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627580.705984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627580.705986:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11915 00000100:00080000:1.0:1778627580.705989:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627580.705991:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627580.705994:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627580.706001:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828533120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627580.706007:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627580.706009:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627580.706015:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627580.706019:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11915 00000100:00080000:1.0:1778627580.706033:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627580.706037:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627580.706041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627580.706045:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627580.706051:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828533248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627580.706059:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627580.706062:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627580.706063:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627580.706066:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11915 00000100:00080000:1.0:1778627580.706068:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627580.706070:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627580.706073:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627580.706084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828533376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627580.706085:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627580/real 1778627580] req@ffff954350195f80 x1865013828532864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627580.706131:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828532864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627580.706139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627580.706142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627580.706145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828532864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627580.706163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627580.706181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627580.706189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627580/real 1778627580] req@ffff954350195c00 x1865013828532992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627580.706197:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828532992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627580.706205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627580.706209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627580.706212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828532992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627580.706218:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627580.706238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627580.706249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627580/real 1778627580] req@ffff954350195180 x1865013828533120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627580.706262:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828533120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627580.706280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627580.706282:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627580.706284:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627580.706289:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627580.706306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627580.706317:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627580/real 1778627580] req@ffff954350197b80 x1865013828533248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627580.706328:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828533248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627580.706351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627580.706355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627580.706359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627580.706365:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627580.706386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627580.706395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627580/real 1778627580] req@ffff954350195880 x1865013828533376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627580.706403:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828533376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627580.706409:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627580.706411:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627580.706419:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627580.897759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627580.897850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627581.665759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627581.921757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627581.921832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627582.689834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627582.945751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627582.945799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627583.713785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627583.969801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627583.969861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627584.737780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627584.993755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627584.993806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627585.761730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627585.825792:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627585.825800:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627585.825804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627585.825811:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11920 00000100:00080000:2.0:1778627585.825821:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627585.825826:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627585.825858:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828533504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627585.825881:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627585.825883:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627585.825885:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627585.825888:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11920 00000100:00080000:2.0:1778627585.825890:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627585.825892:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627585.825899:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627585.825907:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828533632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627585.825914:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627585.825917:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627585.825918:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627585.825920:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11920 00000100:00080000:2.0:1778627585.825923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627585.825925:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627585.825928:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627585.825941:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828533504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627585.825949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828533760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627585.825957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627585.825959:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627585.825960:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627585.825963:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11920 00000100:00080000:2.0:1778627585.825966:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627585.825968:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627585.825972:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627585.825989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828533888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627585.825996:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627585.825998:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627585.825999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627585.826002:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11920 00000100:00080000:2.0:1778627585.826005:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778627585.826007:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627585.826010:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627585.826014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627585.826023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828534016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627585.826055:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627585/real 1778627585] req@ffff954358789180 x1865013828533504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627585.826070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828533504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627585.826078:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627585.826081:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627585.826084:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828533504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627585.826089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627585.826104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627585.826112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627585/real 1778627585] req@ffff954358788a80 x1865013828533632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627585.826120:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828533632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627585.826126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627585.826138:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627585.826141:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627585.826159:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627585.826169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627585.826187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627585/real 1778627585] req@ffff95435878b800 x1865013828533760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627585.826201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828533760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627585.826209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627585.826212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627585.826216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627585.826223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627585.826258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627585.826268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627585/real 1778627585] req@ffff95435878a300 x1865013828533888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627585.826287:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828533888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627585.826293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627585.826295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627585.826297:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828533888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627585.826301:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627585.826315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627585.826324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627585/real 1778627585] req@ffff95435878ad80 x1865013828534016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627585.826337:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828534016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627585.826353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627585.826357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627585.826360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627586.017749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627586.017915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627586.785784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627587.041805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627587.041905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627587.809758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627588.065720:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627588.065850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627588.833761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627589.089792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627589.089876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627589.857730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627590.113746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627590.113808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627590.881761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627590.945771:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627590.945782:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627590.945787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627590.945808:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11925 00000100:00080000:2.0:1778627590.945821:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627590.945827:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627590.945886:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828534144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627590.945913:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627590.945916:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627590.945918:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627590.945921:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11925 00000100:00080000:2.0:1778627590.945924:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627590.945926:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627590.945946:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627590.945954:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828534272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627590.945961:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627590.945964:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627590.945966:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627590.945969:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11925 00000100:00080000:2.0:1778627590.945971:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627590.945974:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627590.945989:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828534144:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778627590.946009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627590.946019:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828534400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627590.946027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627590.946030:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627590.946031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627590.946035:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11925 00000100:00080000:2.0:1778627590.946038:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627590.946041:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627590.946044:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627590.946051:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828534528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627590.946058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627590.946061:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627590.946063:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627590.946065:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11925 00000100:00080000:2.0:1778627590.946067:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627590.946069:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627590.946071:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627590.946076:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627590.946080:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828534656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627590.946149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627590/real 1778627590] req@ffff95435878a680 x1865013828534144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627590.946167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828534144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627590.946176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627590.946194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627590.946198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828534144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627590.946204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627590.946222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627590.946230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627590/real 1778627590] req@ffff954358789f80 x1865013828534272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627590.946238:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828534272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627590.946244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627590.946247:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627590.946249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627590.946278:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627590.946292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627590.946298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627590/real 1778627590] req@ffff95435878ad80 x1865013828534400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627590.946306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828534400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627590.946312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627590.946313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627590.946315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627590.946319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627590.946329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627590.946341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627590/real 1778627590] req@ffff95435878a300 x1865013828534528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627590.946349:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828534528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627590.946355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627590.946357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627590.946359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627590.946362:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627590.946372:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627590.946378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627590/real 1778627590] req@ffff95435878b800 x1865013828534656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627590.946391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828534656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627590.946410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627590.946414:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627590.946417:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627591.137810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627591.137884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627591.905828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627592.161751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627592.161847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627592.929752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627593.185782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627593.185826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627593.953779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627594.209817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627594.209877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627594.977760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627595.233857:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627595.233937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627596.001810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627596.065749:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627596.065763:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627596.065769:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627596.065782:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11930 00000100:00080000:2.0:1778627596.065794:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627596.065801:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627596.065845:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828534784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627596.065874:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627596.065876:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627596.065878:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627596.065882:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11930 00000100:00080000:2.0:1778627596.065884:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627596.065887:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627596.065897:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627596.065906:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828534912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627596.065913:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627596.065916:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627596.065917:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627596.065919:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11930 00000100:00080000:2.0:1778627596.065921:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627596.065943:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627596.065948:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627596.065953:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828534784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627596.065959:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828535040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627596.065967:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627596.065970:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627596.065971:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627596.065973:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11930 00000100:00080000:2.0:1778627596.065977:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627596.065979:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627596.065982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627596.065989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828535168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627596.065996:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627596.065998:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627596.066000:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627596.066002:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11930 00000100:00080000:2.0:1778627596.066004:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627596.066006:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627596.066009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627596.066016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828535296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627596.066043:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627596.066089:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627596/real 1778627596] req@ffff954358788a80 x1865013828534784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627596.066106:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828534784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627596.066116:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627596.066120:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627596.066123:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828534784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627596.066128:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627596.066144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627596.066152:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627596/real 1778627596] req@ffff954358789180 x1865013828534912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627596.066159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828534912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627596.066176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627596.066179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627596.066181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828534912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627596.066204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627596.066218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627596.066224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627596/real 1778627596] req@ffff954358788700 x1865013828535040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627596.066231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828535040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627596.066236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627596.066239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627596.066240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627596.066244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627596.066254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627596.066259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627596/real 1778627596] req@ffff954358789500 x1865013828535168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627596.066265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828535168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627596.066270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627596.066272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627596.066279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627596.066283:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627596.066296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627596.066301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627596/real 1778627596] req@ffff954358789c00 x1865013828535296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627596.066309:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828535296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627596.066315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627596.066317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627596.066318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627596.257757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627596.257867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627597.025788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627597.281781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627597.281842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627598.049770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627598.305776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627598.305898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627599.073787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627599.329764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627599.329833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627600.097789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778627600.353845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627600.353937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627601.121735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627601.185789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627601.185797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627601.185802:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627601.185810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11935 00000100:00080000:0.0:1778627601.185820:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627601.185826:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627601.185858:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828535424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627601.185887:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627601.185890:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627601.185892:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627601.185895:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11935 00000100:00080000:0.0:1778627601.185898:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627601.185919:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828535424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627601.185920:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627601.185943:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627601.185960:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828535552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627601.185988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627601.185991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627601.185993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627601.185996:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11935 00000100:00080000:0.0:1778627601.186003:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627601.186007:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627601.186021:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778627601.186027:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627601.186037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828535680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627601.186047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627601.186050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627601.186052:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627601.186055:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11935 00000100:00080000:0.0:1778627601.186058:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627601.186061:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627601.186066:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627601.186096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828535808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627601.186096:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627601/real 1778627601] req@ffff954342f48e00 x1865013828535424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627601.186122:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627601.186133:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828535424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627601.186135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627601.186139:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627601.186147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627601.186153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778627601.186157:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11935 00000100:00100000:2.0:1778627601.186162:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828535424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627601.186165:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627601.186173:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627601.186179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627601.186193:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627601.186194:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828535936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627601.186236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627601.186248:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627601/real 1778627601] req@ffff954342f4aa00 x1865013828535552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627601.186258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828535552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627601.186264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627601.186266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627601.186268:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627601.186272:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627601.186292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627601.186299:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627601/real 1778627601] req@ffff954342f48380 x1865013828535680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627601.186306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828535680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627601.186312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627601.186314:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627601.186316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627601.186319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627601.186330:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627601.186335:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627601/real 1778627601] req@ffff954342f48000 x1865013828535808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627601.186342:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828535808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627601.186347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627601.186348:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627601.186350:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627601.186361:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627601.186371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627601.186376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627601/real 1778627601] req@ffff954342f49180 x1865013828535936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627601.186393:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828535936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627601.186399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627601.186401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627601.186403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828535936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778627601.377728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627601.377791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627602.145778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627602.402028:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627602.402153:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778627603.169727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627603.425772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627603.425919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627604.193802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627604.449911:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627604.450443:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627605.217757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627605.474055:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627605.474754:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627606.241780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627606.305769:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627606.305778:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627606.305783:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627606.305793:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11940 00000100:00080000:0.0:1778627606.305805:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627606.305812:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627606.305865:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828536064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627606.305890:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627606.305893:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627606.305894:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627606.305897:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11940 00000100:00080000:0.0:1778627606.305900:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627606.305903:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627606.305910:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627606.305920:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828536192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627606.305935:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828536064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627606.305948:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627606.305951:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627606.305952:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627606.305956:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11940 00000100:00080000:0.0:1778627606.305959:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627606.305962:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627606.305966:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627606.305975:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828536320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627606.305982:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627606.305984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627606.305986:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627606.305988:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11940 00000100:00080000:0.0:1778627606.305990:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627606.305994:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627606.305999:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778627606.306001:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627606.306010:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828536448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627606.306017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627606.306019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627606.306021:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627606.306024:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11940 00000100:00080000:0.0:1778627606.306031:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627606.306034:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627606/real 1778627606] req@ffff954342f4bb80 x1865013828536064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627606.306038:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627606.306045:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627606.306054:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828536576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627606.306058:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828536064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627606.306075:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627606.306079:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627606.306082:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828536064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627606.306111:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627606.306128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627606.306136:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627606/real 1778627606] req@ffff954342f49880 x1865013828536192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627606.306144:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828536192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627606.306150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627606.306153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627606.306157:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627606.306163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627606.306181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627606.306192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627606/real 1778627606] req@ffff954342f4a300 x1865013828536320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627606.306205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828536320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627606.306214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627606.306217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627606.306221:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627606.306226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627606.306242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627606.306249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627606/real 1778627606] req@ffff954342f4b480 x1865013828536448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627606.306263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828536448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627606.306268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627606.306270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627606.306272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627606.306276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627606.306287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627606.306293:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627606/real 1778627606] req@ffff954342f49c00 x1865013828536576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627606.306300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828536576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627606.306305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627606.306307:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627606.306308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627606.497790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627606.497956:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627607.265729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627607.521780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627607.522053:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627608.289773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627608.545779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627608.545857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627609.313743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627609.569758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627609.569805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627610.337813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627610.593844:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627610.593907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627611.361837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627611.425801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627611.425814:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627611.425820:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627611.425833:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11945 00000100:00080000:0.0:1778627611.425848:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627611.425856:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627611.425903:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828536704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627611.425928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627611.425931:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627611.425933:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627611.425937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11945 00000100:00080000:0.0:1778627611.425941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627611.425944:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627611.425954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627611.425960:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828536704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627611.425966:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828536832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627611.425974:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627611.425995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627611.425997:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627611.426000:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11945 00000100:00080000:0.0:1778627611.426004:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627611.426007:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627611.426011:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627611.426022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828536960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627611.426029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627611.426031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627611.426032:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778627611.426036:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627611.426037:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11945 00000100:00080000:0.0:1778627611.426042:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627611.426045:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627611.426048:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627611.426057:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828537088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627611.426063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627611.426065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627611.426067:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627611.426070:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11945 00000100:00080000:0.0:1778627611.426073:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627611.426076:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627611.426081:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627611/real 1778627611] req@ffff954342f49500 x1865013828536704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778627611.426086:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627611.426099:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828537216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627611.426106:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828536704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627611.426126:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627611.426134:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627611.426138:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828536704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627611.426164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627611.426188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627611.426202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627611/real 1778627611] req@ffff954342f4ad80 x1865013828536832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627611.426216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828536832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627611.426225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627611.426228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627611.426232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627611.426238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627611.426260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627611.426271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627611/real 1778627611] req@ffff954342f48700 x1865013828536960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627611.426291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828536960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627611.426298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627611.426300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627611.426302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828536960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627611.426307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627611.426323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627611.426329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627611/real 1778627611] req@ffff954342f4b100 x1865013828537088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627611.426337:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828537088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627611.426342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627611.426344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627611.426346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627611.426349:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627611.426359:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627611.426367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627611/real 1778627611] req@ffff954342f49f80 x1865013828537216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627611.426380:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828537216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627611.426400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627611.426405:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627611.426408:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627611.617804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627611.617865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627612.385761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627612.641838:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627612.641846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627613.409799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627613.665744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627613.665815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627614.433764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627614.689786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627614.689842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627615.457785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627615.713739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627615.713813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627616.481757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627616.545771:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627616.545779:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627616.545783:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627616.545791:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11951 00000100:00080000:2.0:1778627616.545801:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627616.545806:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627616.545838:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828537344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627616.545860:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627616.545863:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627616.545865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627616.545868:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11951 00000100:00080000:2.0:1778627616.545872:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627616.545874:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627616.545881:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627616.545892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828537344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627616.545892:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828537472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627616.545912:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627616.545915:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627616.545916:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627616.545919:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11951 00000100:00080000:2.0:1778627616.545923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627616.545925:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627616.545929:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627616.545937:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828537600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627616.545944:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627616.545946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627616.545947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627616.545949:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11951 00000800:00000100:0.0:1778627616.545953:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627616.545958:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627616.545963:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627616.545967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627616.545976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828537728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627616.545982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627616.545985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627616.545987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627616.545989:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627616/real 1778627616] req@ffff954358789880 x1865013828537344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627616.545993:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11951 00000100:00080000:2.0:1778627616.546006:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627616.546011:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627616.546015:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828537344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627616.546019:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627616.546033:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778627616.546038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828537856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627616.546042:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627616.546048:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828537344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627616.546076:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627616.546095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627616.546105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627616/real 1778627616] req@ffff954358788e00 x1865013828537472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627616.546113:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828537472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627616.546129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627616.546132:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627616.546134:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627616.546139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627616.546153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627616.546164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627616/real 1778627616] req@ffff95435878b100 x1865013828537600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627616.546176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828537600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627616.546184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627616.546188:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627616.546191:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627616.546198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627616.546218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627616.546226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627616/real 1778627616] req@ffff95435878bb80 x1865013828537728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627616.546234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828537728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627616.546240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627616.546242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627616.546250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627616.546254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627616.546276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627616.546283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627616/real 1778627616] req@ffff95435878b480 x1865013828537856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627616.546291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828537856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627616.546297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627616.546299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627616.546301:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828537856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627616.737749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627616.737804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627617.505794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627617.761774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627617.761860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627618.529825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627618.785787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627618.785829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627619.553797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627619.809832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627619.809931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627620.577759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627620.833767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627620.833834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627621.601833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627621.665781:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627621.665793:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627621.665799:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627621.665810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11956 00000100:00080000:3.0:1778627621.665825:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627621.665832:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627621.665892:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828537984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627621.665922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627621.665927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627621.665929:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627621.665933:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11956 00000100:00080000:3.0:1778627621.665938:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627621.665943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627621.665955:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627621.665958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828537984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627621.665969:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828538112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627621.665982:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627621.665986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627621.665988:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627621.665993:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11956 00000100:00080000:3.0:1778627621.665997:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627621.666000:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627621.666004:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627621.666016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828538240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627621.666025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627621.666029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627621.666031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627621.666041:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11956 00000100:00080000:3.0:1778627621.666045:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627621.666050:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627621.666056:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627621.666069:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627621.666070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828538368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627621.666083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627621.666087:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627621.666089:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627621.666094:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11956 00000100:00080000:3.0:1778627621.666098:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627621.666102:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627621.666107:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627621.666118:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627621/real 1778627621] req@ffff95434701c000 x1865013828537984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627621.666120:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828538496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627621.666151:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828537984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627621.666161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627621.666165:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627621.666175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828537984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627621.666198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627621.666218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627621.666227:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627621/real 1778627621] req@ffff95434701ea00 x1865013828538112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627621.666236:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828538112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627621.666243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627621.666245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627621.666248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627621.666251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627621.666262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627621.666268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627621/real 1778627621] req@ffff95434701d880 x1865013828538240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627621.666280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828538240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627621.666288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627621.666292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627621.666295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627621.666302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627621.666324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627621.666342:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627621/real 1778627621] req@ffff95434701d180 x1865013828538368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627621.666351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828538368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627621.666357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627621.666360:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627621.666362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627621.666366:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627621.666381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627621.666386:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627621/real 1778627621] req@ffff95434701df80 x1865013828538496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627621.666394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828538496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627621.666400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627621.666402:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627621.666404:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627621.857794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627621.857865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627622.625762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627622.881774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627622.881841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627623.649806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627623.905767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627623.905857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627624.673753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627624.929780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627624.929865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627625.697802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778627625.953849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627625.953856:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627626.721748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627626.785833:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627626.785842:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627626.785846:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627626.785854:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11961 00000100:00080000:2.0:1778627626.785864:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627626.785870:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627626.785900:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828538624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627626.785924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627626.785926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627626.785928:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627626.785944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11961 00000100:00080000:2.0:1778627626.785947:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627626.785950:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627626.785958:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627626.785968:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828538752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627626.785978:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828538624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627626.785978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627626.785986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627626.785988:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627626.785991:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11961 00000100:00080000:2.0:1778627626.785993:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627626.785995:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627626.785998:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627626.786007:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828538880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627626.786013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627626.786016:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627626.786017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627626.786019:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11961 00000100:00080000:2.0:1778627626.786021:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627626.786023:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627626.786027:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627626.786033:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828539008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627626.786037:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627626.786044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627626.786048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627626.786049:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627626.786053:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11961 00000100:00080000:2.0:1778627626.786056:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627626.786058:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627626.786061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627626.786070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828539136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627626.786087:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627626/real 1778627626] req@ffff95435878aa00 x1865013828538624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627626.786102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828538624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627626.786111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627626.786114:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627626.786117:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828538624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627626.786134:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627626.786151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627626.786159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627626/real 1778627626] req@ffff954358788380 x1865013828538752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627626.786167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828538752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627626.786173:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627626.786176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627626.786178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627626.786181:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627626.786191:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627626.786196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627626/real 1778627626] req@ffff95435878b480 x1865013828538880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627626.786204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828538880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627626.786218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627626.786221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627626.786223:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828538880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627626.786227:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627626.786241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627626.786247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627626/real 1778627626] req@ffff95435878bb80 x1865013828539008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627626.786254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828539008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627626.786260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627626.786261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627626.786263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627626.786267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627626.786276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627626.786282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627626/real 1778627626] req@ffff95435878b100 x1865013828539136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627626.786296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828539136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627626.786302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627626.786304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627626.786310:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627626.977756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627626.977800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627627.745767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627628.001776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627628.001840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627628.769744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627629.025771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627629.025826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627629.793781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627630.049774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627630.049828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627630.817799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627631.073762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627631.073820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627631.841754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627631.905800:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627631.905810:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627631.905815:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627631.905824:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11966 00000100:00080000:1.0:1778627631.905835:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627631.905842:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627631.905876:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828539264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627631.905904:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627631.905923:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627631.905926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627631.905931:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11966 00000100:00080000:1.0:1778627631.905936:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627631.905940:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627631.905949:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627631.905954:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828539264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627631.905973:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828539392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627631.905986:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627631.905990:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627631.905992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627631.905995:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11966 00000100:00080000:1.0:1778627631.906000:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627631.906004:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627631.906009:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627631.906009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627631.906023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828539520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627631.906035:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627631.906038:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627631.906041:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627631.906042:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627631/real 1778627631] req@ffff954350196a00 x1865013828539264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627631.906049:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11966 00000100:00080000:1.0:1778627631.906056:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627631.906060:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627631.906065:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828539264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627631.906073:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:0.0:1778627631.906076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627631.906081:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627631.906085:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828539264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627631.906088:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828539648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627631.906100:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627631.906104:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627631.906106:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627631.906107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539392:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627631.906114:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11966 00000100:00080000:1.0:1778627631.906118:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627631.906123:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627631.906128:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627631.906129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627631.906141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627631/real 1778627631] req@ffff954350195500 x1865013828539392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627631.906145:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828539776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627631.906159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828539392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627631.906167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627631.906170:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627631.906182:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627631.906188:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627631.906203:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627631.906209:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627631/real 1778627631] req@ffff954350194e00 x1865013828539520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627631.906217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828539520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627631.906222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627631.906224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627631.906226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627631.906229:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627631.906239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627631.906243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627631/real 1778627631] req@ffff954350194380 x1865013828539648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627631.906250:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828539648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627631.906255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627631.906257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627631.906259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627631.906270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627631.906279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627631.906290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627631/real 1778627631] req@ffff954350194a80 x1865013828539776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627631.906297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828539776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627631.906303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627631.906304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627631.906306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828539776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627632.097765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627632.097823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627632.865788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627633.121751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627633.121827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627633.889754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627634.145782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627634.145853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627634.913800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627635.169780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627635.169849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627635.937800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627636.193825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627636.193907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627636.961740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627637.025812:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627637.025826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627637.025832:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627637.025845:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11971 00000100:00080000:3.0:1778627637.025860:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627637.025869:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627637.025914:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828539904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627637.025965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627637.025968:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627637.025970:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627637.025973:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11971 00000100:00080000:3.0:1778627637.025977:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627637.025980:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627637.025990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627637.026002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828540032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627637.026010:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627637.026012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627637.026014:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627637.026016:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11971 00000100:00080000:3.0:1778627637.026019:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627637.026024:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627637.026025:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828539904:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778627637.026028:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627637.026035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828540160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627637.026045:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627637.026050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627637.026051:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627637.026054:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11971 00000100:00080000:3.0:1778627637.026062:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627637.026065:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627637.026069:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627637.026081:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828540288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627637.026088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627637.026091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627637.026092:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627637.026095:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11971 00000100:00080000:3.0:1778627637.026098:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627637.026100:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627637.026103:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627637.026109:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828540416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627637.026169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627637.026192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627637.026207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627637.026238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627637.217821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627637.217911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627637.985734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627638.241780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627638.241873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627639.009777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627639.265849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627639.265872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627640.033750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627640.097999:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627640.098055:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627640.098070:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627640.098113:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627640.098127:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627640.098142:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627640.098150:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627640.098188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627637/real 1778627640] req@ffff95434701e680 x1865013828540032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627640.098216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828540032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627640.098253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627640.098259:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627640.098264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627640.098272:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627637/real 1778627640] req@ffff95434701f480 x1865013828540160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627640.098288:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828540160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627640.098295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627640.098300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627640.098303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627640.098311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627637/real 1778627640] req@ffff95434701df80 x1865013828540288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627640.098323:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828540288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627640.098332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627640.098335:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627640.098338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627640.098344:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627637/real 1778627640] req@ffff95434701d180 x1865013828540416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627640.098356:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828540416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627640.098372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627640.098376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627640.098378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627640.098384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627637/real 1778627640] req@ffff95434701ca80 x1865013828539904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627640.098396:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828539904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627640.098403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627640.098407:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627640.098410:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828539904:192.168.204.102@tcp:250:kworker.0 00010000:00010000:1.0:1778627640.289831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627640.289866:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627641.057769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627641.313788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627641.313854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627642.081750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627642.145813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627642.145821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627642.145838:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627642.145846:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11976 00000100:00080000:2.0:1778627642.145855:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627642.145861:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627642.145899:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828540544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627642.145924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627642.145926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627642.145928:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627642.145931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11976 00000100:00080000:2.0:1778627642.145933:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627642.145935:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627642.145942:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627642.145950:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828540672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627642.145968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627642.145971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627642.145972:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627642.145974:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11976 00000100:00080000:2.0:1778627642.145976:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627642.145979:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627642.145982:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627642.145991:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828540800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627642.145993:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828540544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627642.146016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627642.146030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627642.146032:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627642.146037:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11976 00000100:00080000:2.0:1778627642.146041:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627642.146046:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627642.146051:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627642.146065:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828540928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627642.146074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627642.146077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627642.146079:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627642.146085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627642.146089:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11976 00000100:00080000:2.0:1778627642.146095:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627642.146099:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627642.146105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627642.146117:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828541056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627642.146146:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627642/real 1778627642] req@ffff954358788e00 x1865013828540544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627642.146169:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828540544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627642.146181:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627642.146186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627642.146190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828540544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627642.146207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627642.146233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627642.146246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627642/real 1778627642] req@ffff954358789880 x1865013828540672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627642.146258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828540672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627642.146266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627642.146270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627642.146274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627642.146301:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627642.146322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627642.146333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627642/real 1778627642] req@ffff954358789c00 x1865013828540800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627642.146344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828540800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627642.146352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627642.146355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627642.146357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627642.146363:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627642.146381:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627642.146392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627642/real 1778627642] req@ffff954358789500 x1865013828540928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627642.146412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828540928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627642.146419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627642.146422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627642.146425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828540928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627642.146431:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627642.146451:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627642.146462:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627642/real 1778627642] req@ffff954358788700 x1865013828541056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627642.146474:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828541056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627642.146483:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627642.146486:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627642.146489:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627642.337741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627642.337777:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627643.105792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627643.361797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627643.361829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627644.129765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627644.385740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627644.385757:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627645.153787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627645.409779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627645.409817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627646.177788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627646.433766:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627646.433771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627647.201799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627647.265736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627647.265744:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627647.265748:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627647.265756:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11981 00000100:00080000:2.0:1778627647.265765:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627647.265771:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627647.265801:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828541184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627647.265823:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627647.265827:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627647.265829:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627647.265833:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11981 00000100:00080000:2.0:1778627647.265836:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627647.265840:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627647.265849:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627647.265862:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828541312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627647.265865:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828541184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627647.265873:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627647.265876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627647.265878:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627647.265881:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11981 00000100:00080000:2.0:1778627647.265884:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627647.265888:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627647.265892:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627647.265914:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828541440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627647.265923:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627647.265926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627647.265927:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627647.265931:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627647.265931:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11981 00000100:00080000:2.0:1778627647.265935:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627647.265939:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627647.265944:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627647.265953:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828541568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627647.265961:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627647.265962:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627647/real 1778627647] req@ffff954358789180 x1865013828541184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627647.265964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627647.265966:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627647.265970:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11981 00000100:00080000:2.0:1778627647.265973:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627647.265976:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627647.265979:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828541184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627647.265980:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627647.265988:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828541696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627647.265990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627647.265994:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627647.265998:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828541184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627647.266035:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627647.266057:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627647.266067:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627647/real 1778627647] req@ffff954358788a80 x1865013828541312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627647.266075:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828541312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627647.266081:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627647.266083:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627647.266085:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627647.266089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627647.266100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627647.266106:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627647/real 1778627647] req@ffff95435878b800 x1865013828541440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627647.266119:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828541440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627647.266124:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627647.266126:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627647.266128:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627647.266132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627647.266144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627647.266150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627647/real 1778627647] req@ffff95435878a300 x1865013828541568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627647.266157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828541568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627647.266162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627647.266163:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627647.266165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627647.266168:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627647.266177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627647.266182:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627647/real 1778627647] req@ffff95435878ad80 x1865013828541696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627647.266189:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828541696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627647.266202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627647.266205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627647.266207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627647.457768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627647.457796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627648.225738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627648.481784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627648.481792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627649.249776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627649.505764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627649.505782:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627650.273734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627650.529740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627650.529763:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627651.297751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627651.553778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627651.553801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627652.321792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627652.385760:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627652.385770:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627652.385774:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627652.385801:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11986 00000100:00080000:3.0:1778627652.385812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627652.385818:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627652.385859:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828541824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627652.385884:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627652.385888:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627652.385890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627652.385894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11986 00000100:00080000:3.0:1778627652.385898:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627652.385902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627652.385912:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627652.385940:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828541952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627652.385943:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828541824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627652.385960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627652.385963:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627652.385965:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627652.385970:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11986 00000100:00080000:3.0:1778627652.385973:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627652.385977:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627652.385982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627652.385995:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828542080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627652.386014:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627652.386025:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627652.386028:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627652.386031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627652.386038:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11986 00000100:00080000:3.0:1778627652.386051:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627652.386055:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627652/real 1778627652] req@ffff95434701d880 x1865013828541824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627652.386060:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627652.386069:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627652.386086:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828541824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627652.386087:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828542208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627652.386119:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627652.386122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778627652.386124:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627652.386128:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627652.386136:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11986 00000100:00080000:3.0:1778627652.386140:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627652.386144:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778627652.386144:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627652.386153:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627652.386155:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828541824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627652.386170:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828542336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627652.386189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627652.386217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627652.386240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627652/real 1778627652] req@ffff95434701ea00 x1865013828541952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627652.386254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828541952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627652.386264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627652.386267:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627652.386271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828541952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627652.386278:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627652.386300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627652.386311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627652/real 1778627652] req@ffff95434701c000 x1865013828542080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627652.386322:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828542080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627652.386329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627652.386333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627652.386337:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627652.386343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627652.386363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627652.386374:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627652/real 1778627652] req@ffff95434701c700 x1865013828542208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627652.386402:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828542208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627652.386418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627652.386422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627652.386426:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627652.386433:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627652.386456:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627652.386468:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627652/real 1778627652] req@ffff95434701f800 x1865013828542336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627652.386480:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828542336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627652.386489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627652.386493:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627652.386496:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627652.577748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627652.577783:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627653.345762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627653.601750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627653.601795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627654.369775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627654.625764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627654.625819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627655.393781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627655.649785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627655.649881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627656.417762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627656.673783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627656.673952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627657.441753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627657.505799:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627657.505845:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627657.505850:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627657.505861:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11992 00000100:00080000:3.0:1778627657.505874:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627657.505881:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627657.505921:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828542464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627657.505952:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627657.505955:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627657.505956:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627657.505960:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11992 00000100:00080000:3.0:1778627657.505963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627657.505966:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627657.505977:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627657.505993:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828542592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627657.506006:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627657.506011:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828542464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627657.506012:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627657.506016:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627657.506022:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11992 00000100:00080000:3.0:1778627657.506027:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627657.506030:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627657.506034:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627657.506045:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828542720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627657.506059:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627657.506061:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627657.506063:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627657.506067:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11992 00000100:00080000:3.0:1778627657.506070:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627657.506073:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627657.506077:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627657.506095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627657.506098:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828542848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627657.506106:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627657.506110:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627657.506111:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627657.506115:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11992 00000100:00080000:3.0:1778627657.506118:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627657.506121:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627657.506124:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627657.506131:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828542976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627657.506140:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627657/real 1778627657] req@ffff95434701f100 x1865013828542464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627657.506157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828542464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627657.506181:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627657.506185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627657.506188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828542464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627657.506212:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627657.506230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627657.506239:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627657/real 1778627657] req@ffff95434701c380 x1865013828542592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627657.506248:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828542592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627657.506255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627657.506257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627657.506259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627657.506263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627657.506274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627657.506279:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627657/real 1778627657] req@ffff95434701d500 x1865013828542720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627657.506286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828542720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627657.506292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627657.506294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627657.506295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627657.506305:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627657.506319:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627657.506325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627657/real 1778627657] req@ffff95434701f800 x1865013828542848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627657.506333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828542848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627657.506338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627657.506340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627657.506342:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627657.506352:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627657.506363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627657.506369:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627657/real 1778627657] req@ffff95434701c700 x1865013828542976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627657.506377:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828542976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627657.506382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627657.506384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627657.506386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828542976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627657.697787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627657.697871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627658.465813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627658.721867:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627658.721926:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627659.489746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627659.745819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627659.745845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627660.513773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627660.769849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627660.769944:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627661.537800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627661.793828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627661.793866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627662.561826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627662.625869:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627662.625882:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627662.625887:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627662.625898:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 11997 00000100:00080000:2.0:1778627662.625913:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627662.625920:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627662.625983:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828543104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627662.626037:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627662.626047:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627662.626049:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627662.626071:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11997 00000100:00080000:2.0:1778627662.626075:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627662.626079:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627662.626090:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627662.626118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828543104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627662.626132:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828543232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627662.626142:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627662.626144:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627662.626146:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627662.626148:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11997 00000100:00080000:2.0:1778627662.626152:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627662.626155:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627662.626159:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627662.626170:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828543360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627662.626176:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627662.626179:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627662.626180:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627662.626184:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11997 00000100:00080000:2.0:1778627662.626186:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627662.626188:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627662.626191:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627662.626198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627662.626201:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828543488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627662.626209:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627662.626212:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627662.626214:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627662.626222:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 11997 00000100:00080000:2.0:1778627662.626226:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627662.626230:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627662.626235:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627662.626243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627662/real 1778627662] req@ffff954358789f80 x1865013828543104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627662.626246:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828543616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627662.626274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828543104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627662.626284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627662.626288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627662.626291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828543104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627662.626313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627662.626329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627662.626338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627662/real 1778627662] req@ffff95435878a680 x1865013828543232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627662.626346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828543232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627662.626352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627662.626355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627662.626357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627662.626360:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627662.626370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627662.626375:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627662/real 1778627662] req@ffff95435878ad80 x1865013828543360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627662.626391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828543360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627662.626397:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627662.626399:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627662.626401:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627662.626405:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627662.626418:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627662.626424:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627662/real 1778627662] req@ffff95435878a300 x1865013828543488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627662.626432:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828543488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627662.626437:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627662.626439:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627662.626441:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627662.626444:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627662.626454:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627662.626459:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627662/real 1778627662] req@ffff95435878b800 x1865013828543616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627662.626466:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828543616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627662.626481:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627662.626490:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627662.626492:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627662.817812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627662.817880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627663.585837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627663.841797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627663.841805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627664.609816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627664.865852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627664.865916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627665.633792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627665.889778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627665.889866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627666.657790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627666.913781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627666.913822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627667.681790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627667.745809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627667.745818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627667.745822:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627667.745830:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12002 00000100:00080000:0.0:1778627667.745840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627667.745846:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627667.745879:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828543744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627667.745906:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627667.745909:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627667.745910:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627667.745913:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12002 00000100:00080000:0.0:1778627667.745916:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627667.745919:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627667.745927:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627667.745936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828543872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627667.745962:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828543744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627667.745972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627667.745975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627667.745976:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627667.745982:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12002 00000100:00080000:0.0:1778627667.745986:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627667.745988:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627667.745994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627667.746005:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828544000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627667.746012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627667.746014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627667.746015:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627667.746018:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12002 00000100:00080000:0.0:1778627667.746021:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627667.746023:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627667.746027:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627667.746032:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828544128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627667.746039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627667.746042:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627667.746044:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627667.746048:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12002 00000100:00080000:0.0:1778627667.746059:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627667.746066:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627667.746071:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627667.746078:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627667.746095:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828544256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627667.746113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627667/real 1778627667] req@ffff954342f48a80 x1865013828543744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627667.746135:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828543744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627667.746145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627667.746149:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627667.746152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828543744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627667.746178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627667.746196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627667.746204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627667/real 1778627667] req@ffff954342f4a680 x1865013828543872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627667.746230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828543872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627667.746236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627667.746238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627667.746240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828543872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627667.746244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627667.746258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627667.746264:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627667/real 1778627667] req@ffff954342f49f80 x1865013828544000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627667.746271:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828544000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627667.746276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627667.746278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627667.746280:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627667.746284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627667.746295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627667.746300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627667/real 1778627667] req@ffff954342f4b100 x1865013828544128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627667.746307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828544128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627667.746323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627667.746326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627667.746328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627667.746332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627667.746345:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627667.746351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627667/real 1778627667] req@ffff954342f48700 x1865013828544256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627667.746359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828544256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627667.746370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627667.746373:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627667.746374:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627667.937834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627667.937891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627668.705778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627668.962282:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627668.962717:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627669.729744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627669.985798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627669.986331:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627670.753774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627671.009983:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627671.010189:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627671.777882:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627672.034009:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627672.034041:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627672.801767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627672.865853:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627672.865864:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627672.865869:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627672.865881:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12007 00000100:00080000:2.0:1778627672.865895:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627672.865901:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627672.865967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828544384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627672.865997:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627672.866001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627672.866002:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627672.866006:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12007 00000100:00080000:2.0:1778627672.866010:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627672.866013:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627672.866022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627672.866031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828544512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627672.866040:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828544384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627672.866041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627672.866049:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627672.866050:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627672.866054:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12007 00000100:00080000:2.0:1778627672.866056:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627672.866058:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627672.866061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627672.866068:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828544640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627672.866077:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627672.866081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627672.866083:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627672.866094:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12007 00000100:00080000:2.0:1778627672.866100:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627672.866105:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627672.866111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627672.866116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627672.866129:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828544768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627672.866141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627672.866146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627672.866148:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627672.866153:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12007 00000100:00080000:2.0:1778627672.866157:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627672.866161:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627672.866166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627672.866167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627672/real 1778627672] req@ffff954358788a80 x1865013828544384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627672.866178:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828544896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627672.866188:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828544384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627672.866198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627672.866202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627672.866206:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828544384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627672.866230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627672.866247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627672.866256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627672/real 1778627672] req@ffff954358789180 x1865013828544512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627672.866281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828544512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627672.866290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627672.866294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627672.866297:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627672.866304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627672.866327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627672.866338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627672/real 1778627672] req@ffff954358788700 x1865013828544640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627672.866347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828544640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627672.866353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627672.866355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627672.866357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627672.866361:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627672.866375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627672.866382:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627672/real 1778627672] req@ffff954358789500 x1865013828544768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627672.866389:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828544768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627672.866406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627672.866409:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627672.866411:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627672.866415:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627672.866428:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627672.866434:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627672/real 1778627672] req@ffff954358789c00 x1865013828544896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627672.866442:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828544896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627672.866448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627672.866450:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627672.866451:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828544896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627673.057819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627673.058082:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627673.825796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627674.081831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778627674.082079:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627674.849791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627675.105773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627675.105820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627675.873786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627676.129826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627676.129903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627676.897766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627677.153784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627677.153887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627677.921795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627677.985774:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627677.985786:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627677.985791:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627677.985801:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12012 00000100:00080000:2.0:1778627677.985813:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627677.985820:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627677.985876:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828545024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627677.985904:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627677.985907:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627677.985908:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627677.985912:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12012 00000100:00080000:2.0:1778627677.985915:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627677.985917:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627677.985928:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627677.985938:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828545152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627677.985946:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627677.985949:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627677.985952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627677.985959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828545024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627677.985959:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12012 00000100:00080000:2.0:1778627677.985963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627677.985973:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627677.985978:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627677.985992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828545280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627677.986001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627677.986005:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627677.986007:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627677.986020:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12012 00000100:00080000:2.0:1778627677.986026:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627677.986031:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627677.986038:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627677.986046:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627677.986055:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828545408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627677.986068:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627677.986072:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627677.986074:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627677.986079:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12012 00000100:00080000:2.0:1778627677.986084:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627677.986086:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627677/real 1778627677] req@ffff954358789880 x1865013828545024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627677.986094:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627677.986108:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627677.986123:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828545024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627677.986129:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828545536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627677.986168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627677.986174:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627677.986179:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828545024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627677.986208:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627677.986230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627677.986242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627677/real 1778627677] req@ffff954358788e00 x1865013828545152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627677.986254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828545152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627677.986263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627677.986266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627677.986270:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627677.986276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627677.986293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627677.986300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627677/real 1778627677] req@ffff95435878b100 x1865013828545280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627677.986312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828545280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627677.986320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627677.986323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627677.986326:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627677.986338:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627677.986355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627677.986364:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627677/real 1778627677] req@ffff95435878bb80 x1865013828545408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627677.986376:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828545408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627677.986384:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627677.986387:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627677.986390:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627677.986396:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627677.986410:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627677.986438:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627677/real 1778627677] req@ffff95435878b480 x1865013828545536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627677.986450:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828545536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627677.986460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627677.986463:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627677.986467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627678.177775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627678.177868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627678.945746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627679.201774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627679.201839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627679.969725:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627680.225775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627680.225863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627680.993765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627681.249773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627681.249881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627682.017786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627682.273773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627682.273834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627683.041786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627683.105737:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627683.105744:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627683.105748:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627683.105756:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12017 00000100:00080000:2.0:1778627683.105766:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627683.105786:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627683.105814:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828545664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627683.105840:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627683.105843:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627683.105844:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627683.105847:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12017 00000100:00080000:2.0:1778627683.105850:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627683.105854:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627683.105861:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627683.105870:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828545792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627683.105878:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627683.105881:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627683.105883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627683.105884:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828545664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627683.105887:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12017 00000100:00080000:2.0:1778627683.105892:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627683.105895:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627683.105899:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627683.105908:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828545920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627683.105914:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627683.105917:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627683.105923:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627683.105925:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12017 00000100:00080000:2.0:1778627683.105929:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627683.105932:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627683.105936:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627683.105943:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627683.105952:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828546048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627683.105960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627683.105962:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627683.105963:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627683.105967:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12017 00000100:00080000:2.0:1778627683.105970:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627683.105976:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627683.105977:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627683/real 1778627683] req@ffff954358788380 x1865013828545664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627683.105979:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627683.105990:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828546176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627683.105998:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828545664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627683.106007:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627683.106011:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627683.106015:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828545664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627683.106032:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627683.106068:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627683.106077:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627683/real 1778627683] req@ffff95435878aa00 x1865013828545792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627683.106085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828545792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627683.106091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627683.106094:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627683.106096:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627683.106100:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627683.106112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627683.106133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627683/real 1778627683] req@ffff95435878b480 x1865013828545920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627683.106148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828545920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627683.106154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627683.106156:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627683.106158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828545920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627683.106161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627683.106174:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627683.106179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627683/real 1778627683] req@ffff95435878bb80 x1865013828546048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627683.106205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828546048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627683.106211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627683.106214:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627683.106216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627683.106220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627683.106233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627683.106240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627683/real 1778627683] req@ffff95435878b100 x1865013828546176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627683.106249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828546176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627683.106267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627683.106285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627683.106287:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627683.297741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627683.297857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627684.065786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627684.321768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627684.321854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627685.089743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627685.345772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627685.345831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627686.113724:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627686.369783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627686.369820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627687.137745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627687.393741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627687.393817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627688.161754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627688.225800:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627688.225812:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627688.225816:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627688.225826:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12022 00000100:00080000:3.0:1778627688.225853:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627688.225861:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627688.225895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828546304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627688.225925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627688.225929:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627688.225931:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627688.225935:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12022 00000100:00080000:3.0:1778627688.225939:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627688.225942:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627688.225952:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627688.225967:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828546432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627688.225969:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828546304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627688.225985:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627688.225989:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627688.225991:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627688.225996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12022 00000100:00080000:3.0:1778627688.226001:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627688.226005:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627688.226010:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627688.226023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828546560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627688.226025:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627688.226037:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627688.226050:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627688.226053:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627688.226056:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627688/real 1778627688] req@ffff95434701c000 x1865013828546304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627688.226061:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12022 00000100:00080000:3.0:1778627688.226070:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627688.226077:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627688.226081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828546304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778627688.226085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:0.0:1778627688.226099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778627688.226105:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828546688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627688.226107:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627688.226117:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828546304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627688.226128:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627688.226131:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627688.226134:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627688.226139:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12022 00000100:00080000:3.0:1778627688.226144:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627688.226148:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546432:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778627688.226150:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627688.226168:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627688.226172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627688.226184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627688/real 1778627688] req@ffff95434701ea00 x1865013828546432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627688.226190:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828546816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627688.226201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828546432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627688.226228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627688.226231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627688.226234:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627688.226240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627688.226257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627688.226265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627688/real 1778627688] req@ffff95434701d880 x1865013828546560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627688.226272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828546560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627688.226278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627688.226280:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627688.226282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627688.226286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627688.226296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627688.226301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627688/real 1778627688] req@ffff95434701ca80 x1865013828546688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627688.226308:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828546688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627688.226313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627688.226315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627688.226323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627688.226335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627688.226352:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627688.226363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627688/real 1778627688] req@ffff95434701d180 x1865013828546816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627688.226375:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828546816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627688.226385:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627688.226388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627688.226392:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828546816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627688.417737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627688.417789:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627689.185797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627689.441774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627689.441837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627690.209757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627690.465800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627690.465849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627691.233788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627691.489739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627691.489814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627692.257758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627692.513737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627692.513803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627693.281788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627693.345743:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627693.345751:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627693.345770:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627693.345778:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12027 00000100:00080000:3.0:1778627693.345789:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627693.345795:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627693.345824:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828546944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627693.345848:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627693.345852:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627693.345854:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627693.345858:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12027 00000100:00080000:3.0:1778627693.345862:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627693.345866:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627693.345875:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627693.345890:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828547072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627693.345894:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828546944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627693.345910:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627693.345914:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627693.345915:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627693.345919:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12027 00000100:00080000:3.0:1778627693.345923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627693.345926:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627693.345931:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627693.345940:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828547200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627693.345957:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627693.345962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627693.345964:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627693.345966:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627693.345971:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12027 00000100:00080000:3.0:1778627693.345986:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627693.345991:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627693/real 1778627693] req@ffff95434701df80 x1865013828546944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627693.345995:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627693.346006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627693.346014:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828546944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627693.346023:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778627693.346025:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828547328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627693.346029:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778627693.346036:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627693.346041:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828546944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627693.346043:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627693.346047:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627693.346054:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12027 00000100:00080000:3.0:1778627693.346058:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627693.346062:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627693.346068:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627693.346072:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778627693.346084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828547456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627693.346104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627693.346131:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627693/real 1778627693] req@ffff95434701f480 x1865013828547072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627693.346141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828547072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627693.346147:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627693.346150:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627693.346152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627693.346156:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627693.346170:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627693.346176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627693/real 1778627693] req@ffff95434701e680 x1865013828547200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627693.346183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828547200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627693.346188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627693.346190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627693.346192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627693.346196:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627693.346206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627693.346218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627693/real 1778627693] req@ffff95434701d180 x1865013828547328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627693.346225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828547328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627693.346231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627693.346233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627693.346235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627693.346246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627693.346257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627693.346262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627693/real 1778627693] req@ffff95434701ca80 x1865013828547456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627693.346269:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828547456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627693.346275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627693.346277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627693.346279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627693.537750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627693.537813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627694.305745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627694.561758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627694.561794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627695.329747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627695.585769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627695.585817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627696.353741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627696.609786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627696.609855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627697.377827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627697.633831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627697.633910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627698.401780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627698.465742:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627698.465750:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627698.465754:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627698.465762:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12032 00000100:00080000:3.0:1778627698.465771:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627698.465777:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627698.465809:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828547584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627698.465832:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627698.465834:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627698.465836:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627698.465839:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12032 00000100:00080000:3.0:1778627698.465841:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627698.465843:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627698.465850:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627698.465858:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828547712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627698.465864:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627698.465867:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627698.465868:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627698.465870:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12032 00000100:00080000:3.0:1778627698.465872:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627698.465874:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627698.465876:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627698.465899:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828547840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627698.465904:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828547584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627698.465912:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627698.465915:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627698.465916:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627698.465919:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12032 00000100:00080000:3.0:1778627698.465922:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627698.465925:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627698.465928:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627698.465938:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828547968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627698.465944:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627698.465946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627698.465947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627698.465949:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12032 00000100:00080000:3.0:1778627698.465951:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627698.465954:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627698.465957:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627698.465963:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627698.465969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828548096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627698.466033:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627698/real 1778627698] req@ffff95434701d880 x1865013828547584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627698.466060:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828547584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627698.466068:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627698.466071:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627698.466074:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828547584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627698.466079:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627698.466095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627698.466103:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627698/real 1778627698] req@ffff95434701ea00 x1865013828547712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627698.466111:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828547712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627698.466116:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627698.466118:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627698.466121:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627698.466152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627698.466172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627698.466183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627698/real 1778627698] req@ffff95434701c000 x1865013828547840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627698.466196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828547840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627698.466211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627698.466213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627698.466216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627698.466220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627698.466235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627698.466242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627698/real 1778627698] req@ffff95434701c700 x1865013828547968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627698.466249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828547968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627698.466254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627698.466256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627698.466258:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828547968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627698.466261:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627698.466269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627698.466274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627698/real 1778627698] req@ffff95434701f800 x1865013828548096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627698.466281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828548096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627698.466286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627698.466288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627698.466293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627698.657741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627698.657800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627699.425798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627699.681779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627699.681867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627700.449769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627700.705727:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627700.705823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627701.473778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627701.729802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627701.729870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627702.497766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627702.753758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627702.753817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627703.521780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627703.585780:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627703.585788:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627703.585804:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627703.585813:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12038 00000100:00080000:2.0:1778627703.585822:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627703.585829:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627703.585879:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828548224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627703.585921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627703.585925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627703.585927:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627703.585932:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12038 00000100:00080000:2.0:1778627703.585935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627703.585938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627703.585947:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627703.585959:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828548352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627703.585967:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828548224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627703.585989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627703.585992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627703.585994:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627703.585999:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12038 00000100:00080000:2.0:1778627703.586004:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627703.586008:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627703.586013:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627703.586036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828548480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627703.586050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627703.586054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1778627703.586057:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548352:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778627703.586058:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627703.586067:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12038 00000100:00080000:2.0:1778627703.586073:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627703.586078:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627703.586082:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548480:192.168.204.102@tcp:38:kworker.0 00000008:00100000:2.0:1778627703.586084:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627703.586099:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828548608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627703.586113:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627703.586115:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627703.586117:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627703.586122:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548608:192.168.204.102@tcp:8:kworker.0 00000100:00080000:2.0:1778627703.586122:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12038 00000100:00080000:2.0:1778627703.586132:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627703.586137:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627703.586142:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627703.586155:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828548736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627703.586183:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627703.777768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627703.777830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627704.545776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627704.801773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627704.801835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627705.569775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627705.825767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627705.825831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627706.593751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627706.658019:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627706.658069:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627706.658084:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627706.658127:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627706.658146:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627706.658153:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627706.658161:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627706.658194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627703/real 1778627706] req@ffff954358788700 x1865013828548736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627706.658237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828548736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627706.658254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627706.658261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627706.658266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627706.658276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627703/real 1778627706] req@ffff954358789500 x1865013828548608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627706.658315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828548608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627706.658333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627706.658338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627706.658343:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627706.658357:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627703/real 1778627706] req@ffff954358789880 x1865013828548352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627706.658381:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828548352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627706.658392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627706.658397:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627706.658401:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627706.658409:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627703/real 1778627706] req@ffff954358789c00 x1865013828548480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627706.658423:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828548480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627706.658431:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627706.658435:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627706.658438:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627706.658446:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627703/real 1778627706] req@ffff954358788e00 x1865013828548224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627706.658475:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828548224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627706.658485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627706.658490:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627706.658495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828548224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627706.849743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627706.849822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627707.617751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627707.873787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627707.873857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627708.641742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627708.705765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627708.705775:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627708.705797:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627708.705828:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12043 00000100:00080000:2.0:1778627708.705840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627708.705867:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627708.705918:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828548864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627708.705967:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627708.705974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627708.705977:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627708.706001:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12043 00000100:00080000:2.0:1778627708.706007:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627708.706027:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627708.706041:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627708.706065:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828548992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627708.706077:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828548864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627708.706107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627708.706112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627708.706115:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627708.706124:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12043 00000100:00080000:2.0:1778627708.706132:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627708.706139:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627708.706147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778627708.706186:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627708.706187:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828549120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627708.706208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627708.706215:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627708.706218:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627708.706226:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12043 00000100:00080000:2.0:1778627708.706233:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627708.706240:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627708.706249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627708/real 1778627708] req@ffff954358789180 x1865013828548864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627708.706252:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627708.706276:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828549248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627708.706291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828548864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627708.706297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627708.706312:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627708.706317:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627708.706328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627708.706329:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12043 00000100:00080000:0.0:1778627708.706339:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627708.706352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828548864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627708.706352:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627708.706366:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627708.706375:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627708.706395:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828549376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627708.706428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627708.706472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627708.706495:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627708/real 1778627708] req@ffff954358788a80 x1865013828548992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627708.706517:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828548992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627708.706546:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627708.706552:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627708.706558:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828548992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627708.706569:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627708.706608:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627708.706629:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627708/real 1778627708] req@ffff95435878b800 x1865013828549120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627708.706646:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828549120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627708.706659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627708.706664:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627708.706722:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627708.706734:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627708.706770:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627708.706793:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627708/real 1778627708] req@ffff95435878a300 x1865013828549248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627708.706834:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828549248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627708.706846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627708.706852:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627708.706869:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627708.706880:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627708.706916:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627708.706934:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627708/real 1778627708] req@ffff95435878ad80 x1865013828549376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627708.706953:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828549376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627708.706967:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627708.706972:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627708.706977:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627708.897779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627708.897861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627709.665746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627709.921732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627709.921817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627710.689791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627710.945731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627710.945798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627711.713765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627711.969805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627711.969867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627712.737823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627712.993740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627712.993843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627713.761791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627713.825815:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627713.825824:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627713.825828:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627713.825836:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12048 00000100:00080000:1.0:1778627713.825846:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627713.825864:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627713.825895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828549504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627713.825919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627713.825922:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627713.825924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627713.825927:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12048 00000100:00080000:1.0:1778627713.825931:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627713.825933:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627713.825940:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627713.825949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828549632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627713.825958:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627713.825960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627713.825962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627713.825964:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12048 00000100:00080000:1.0:1778627713.825966:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627713.825968:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627713.825972:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627713.825976:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828549504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627713.825982:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828549760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627713.826006:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627713.826009:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627713.826017:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627713.826021:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12048 00000100:00080000:1.0:1778627713.826025:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627713.826028:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627713.826034:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627713.826043:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828549888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627713.826051:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627713.826054:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627713.826056:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627713.826062:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627713.826064:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12048 00000100:00080000:1.0:1778627713.826069:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627713.826072:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627713.826076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627713.826085:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828550016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627713.826113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627713/real 1778627713] req@ffff954350194000 x1865013828549504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627713.826139:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828549504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627713.826150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627713.826153:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627713.826165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828549504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627713.826172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627713.826195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627713.826206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627713/real 1778627713] req@ffff954350194700 x1865013828549632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627713.826216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828549632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627713.826223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627713.826226:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627713.826230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627713.826273:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627713.826296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627713.826309:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627713/real 1778627713] req@ffff954350197480 x1865013828549760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627713.826324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828549760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627713.826332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627713.826335:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627713.826338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627713.826343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627713.826361:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627713.826394:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627713/real 1778627713] req@ffff954350196d80 x1865013828549888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627713.826404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828549888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627713.826410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627713.826412:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627713.826415:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828549888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627713.826419:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627713.826432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627713.826439:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627713/real 1778627713] req@ffff954350197800 x1865013828550016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627713.826448:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828550016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627713.826454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627713.826456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627713.826458:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627714.017764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627714.017821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627714.785828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627715.041744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627715.041817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627715.809747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627716.065763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627716.065828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627716.833780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627717.089784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627717.089856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627717.857808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627718.113772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627718.113824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627718.881801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627718.945783:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627718.945806:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627718.945811:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627718.945818:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12053 00000100:00080000:3.0:1778627718.945828:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627718.945851:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627718.945879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828550144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627718.945903:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627718.945906:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627718.945907:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627718.945910:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12053 00000100:00080000:3.0:1778627718.945913:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627718.945915:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627718.945930:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627718.945966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828550272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627718.945979:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627718.945984:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778627718.945986:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828550144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627718.945987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627718.945996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12053 00000100:00080000:3.0:1778627718.946001:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627718.946013:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627718.946019:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627718.946032:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828550400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627718.946039:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627718.946042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627718.946048:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627718.946051:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12053 00000100:00080000:3.0:1778627718.946055:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627718.946058:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627718.946062:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627718.946079:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627718.946083:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828550528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627718.946092:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627718.946095:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627718.946096:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627718.946098:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12053 00000100:00080000:3.0:1778627718.946100:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627718.946102:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627718.946105:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627718.946113:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828550656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627718.946147:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627718/real 1778627718] req@ffff95434701d500 x1865013828550144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627718.946169:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828550144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627718.946181:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627718.946186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627718.946190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828550144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627718.946213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627718.946238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627718.946251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627718/real 1778627718] req@ffff95434701c380 x1865013828550272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627718.946264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828550272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627718.946273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627718.946277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627718.946281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627718.946288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627718.946308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627718.946319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627718/real 1778627718] req@ffff95434701f100 x1865013828550400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627718.946332:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828550400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627718.946353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627718.946358:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627718.946362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627718.946368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627718.946391:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627718.946402:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627718/real 1778627718] req@ffff95434701f800 x1865013828550528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627718.946415:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828550528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627718.946424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627718.946428:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627718.946431:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627718.946437:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627718.946456:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627718.946466:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627718/real 1778627718] req@ffff95434701c700 x1865013828550656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627718.946479:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828550656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627718.946488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627718.946492:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627718.946517:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627719.137770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627719.137801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627719.905781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627720.161745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627720.161808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627720.929743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627721.185778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627721.185870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627721.953740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778627722.209756:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778627722.209767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627722.977776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627723.233780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627723.233804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627724.001748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627724.065746:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627724.065754:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627724.065758:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627724.065765:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12058 00000100:00080000:3.0:1778627724.065792:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627724.065798:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627724.065836:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828550784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627724.065859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627724.065862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627724.065863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627724.065867:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12058 00000100:00080000:3.0:1778627724.065869:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627724.065872:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627724.065879:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627724.065889:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828550912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627724.065896:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627724.065898:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627724.065900:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627724.065902:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12058 00000100:00080000:3.0:1778627724.065905:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627724.065909:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828550784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627724.065910:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627724.065917:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627724.065925:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828551040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627724.065935:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627724.065945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627724.065947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627724.065951:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12058 00000100:00080000:3.0:1778627724.065956:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627724.065961:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627724.065967:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778627724.065967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627724.065985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828551168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627724.065995:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627724.065998:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627724.066001:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627724/real 1778627724] req@ffff95434701c000 x1865013828550784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627724.066002:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627724.066009:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12058 00000100:00080000:3.0:1778627724.066014:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627724.066017:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627724.066020:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627724.066035:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828551296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627724.066039:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828550784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627724.066058:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627724.066063:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627724.066066:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828550784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627724.066074:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627724.066091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627724.066100:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627724/real 1778627724] req@ffff95434701ea00 x1865013828550912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627724.066108:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828550912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627724.066125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627724.066127:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627724.066130:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828550912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627724.066148:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627724.066162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627724.066169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627724/real 1778627724] req@ffff95434701d880 x1865013828551040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627724.066176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828551040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627724.066182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627724.066184:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627724.066186:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627724.066189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627724.066199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627724.066204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627724/real 1778627724] req@ffff95434701ca80 x1865013828551168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627724.066211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828551168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627724.066216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627724.066218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627724.066225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627724.066229:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627724.066241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627724.066247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627724/real 1778627724] req@ffff95434701d180 x1865013828551296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627724.066254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828551296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627724.066259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627724.066261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627724.066263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627724.257756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627724.257807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627725.025798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627725.281799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627725.281821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627726.049746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627726.305760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627726.305780:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627727.073773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627727.329784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627727.329832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627728.097752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627728.353759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627728.353879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627729.121761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627729.185775:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627729.185786:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627729.185806:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627729.185815:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12063 00000100:00080000:3.0:1778627729.185842:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627729.185848:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627729.185878:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828551424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627729.185899:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627729.185902:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627729.185903:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627729.185907:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12063 00000100:00080000:3.0:1778627729.185924:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627729.185927:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627729.185935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627729.185939:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828551424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627729.185948:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828551552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627729.185956:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627729.185958:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627729.185960:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627729.185962:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12063 00000100:00080000:3.0:1778627729.185965:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627729.185967:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627729.185971:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627729.185992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828551680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627729.186004:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627729.186025:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627729.186030:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627729.186032:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627729.186034:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627729/real 1778627729] req@ffff95434701e680 x1865013828551424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627729.186041:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12063 00000100:00080000:3.0:1778627729.186055:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627729.186060:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627729.186065:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828551424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778627729.186065:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627729.186082:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828551808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627729.186110:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627729.186111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778627729.186115:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627729.186117:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627729.186119:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778627729.186122:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12063 00000100:00080000:3.0:1778627729.186125:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627729.186126:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828551424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627729.186130:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627729.186134:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627729.186142:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828551936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627729.186150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627729.186170:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627729.186179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627729/real 1778627729] req@ffff95434701f480 x1865013828551552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627729.186187:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828551552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627729.186192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627729.186195:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627729.186197:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627729.186200:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627729.186209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627729.186214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627729/real 1778627729] req@ffff95434701df80 x1865013828551680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627729.186221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828551680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627729.186237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627729.186239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627729.186241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627729.186249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627729.186262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627729.186268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627729/real 1778627729] req@ffff95434701d180 x1865013828551808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627729.186275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828551808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627729.186280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627729.186281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627729.186283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627729.186294:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627729.186303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627729.186308:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627729/real 1778627729] req@ffff95434701ca80 x1865013828551936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627729.186315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828551936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627729.186320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627729.186322:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627729.186323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828551936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627729.377776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627729.377844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627730.145781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627730.401735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627730.401882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627731.169770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627731.425763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627731.425869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627732.193743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627732.449772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627732.449836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627733.217752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627733.473801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627733.473911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627734.241808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627734.305781:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627734.305789:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627734.305793:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627734.305801:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12068 00000100:00080000:3.0:1778627734.305811:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627734.305819:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627734.305859:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828552064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627734.305892:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627734.305895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627734.305896:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627734.305900:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12068 00000100:00080000:3.0:1778627734.305903:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627734.305906:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627734.305914:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627734.305925:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828552192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627734.305933:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627734.305937:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778627734.305940:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828552064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627734.305945:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627734.305950:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12068 00000100:00080000:3.0:1778627734.305953:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627734.305956:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627734.305959:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627734.305984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828552320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627734.305991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627734.305993:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627734.305995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627734.305998:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12068 00000100:00080000:3.0:1778627734.306001:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627734.306004:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627734.306008:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627734.306013:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627734.306018:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828552448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627734.306026:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627734.306029:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627734.306030:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627734.306033:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12068 00000100:00080000:3.0:1778627734.306035:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627734.306038:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627734.306041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627734.306050:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627734/real 1778627734] req@ffff95434701d880 x1865013828552064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627734.306056:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828552576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627734.306069:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828552064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627734.306078:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627734.306082:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627734.306085:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828552064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627734.306104:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627734.306120:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627734.306128:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627734/real 1778627734] req@ffff95434701ea00 x1865013828552192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627734.306136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828552192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627734.306142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627734.306144:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627734.306147:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627734.306150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627734.306160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627734.306169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627734/real 1778627734] req@ffff95434701c000 x1865013828552320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627734.306191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828552320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627734.306200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627734.306203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627734.306207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627734.306214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627734.306233:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627734.306241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627734/real 1778627734] req@ffff95434701c700 x1865013828552448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627734.306249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828552448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627734.306254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627734.306257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627734.306259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627734.306262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627734.306273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627734.306278:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627734/real 1778627734] req@ffff95434701f800 x1865013828552576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627734.306285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828552576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627734.306313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627734.306317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627734.306321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627734.497821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627734.497850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627735.265764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627735.521888:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627735.521941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627736.289789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627736.545958:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627736.546403:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627737.313761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627737.569767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627737.569927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627738.337781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627738.593843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627738.593995:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627739.361746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627739.425781:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627739.425791:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627739.425797:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627739.425806:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12073 00000100:00080000:3.0:1778627739.425817:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627739.425824:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627739.425857:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828552704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627739.425886:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627739.425890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627739.425892:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627739.425896:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12073 00000100:00080000:3.0:1778627739.425899:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627739.425903:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627739.425913:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627739.425925:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828552832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627739.425931:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828552704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627739.425944:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627739.425947:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627739.425949:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627739.425954:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12073 00000100:00080000:3.0:1778627739.425958:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627739.425962:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627739.425981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778627739.425990:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627739.425996:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828552960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627739.426008:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627739.426012:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627739.426014:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627739.426019:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12073 00000100:00080000:3.0:1778627739.426023:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627739.426025:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627739/real 1778627739] req@ffff95434701f100 x1865013828552704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627739.426031:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627739.426041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627739.426047:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828552704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627739.426055:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828553088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627739.426057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627739.426064:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627739.426071:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828552704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627739.426074:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627739.426078:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627739.426079:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627739.426084:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12073 00000100:00080000:3.0:1778627739.426088:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627739.426092:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552832:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778627739.426094:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627739.426103:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627739.426112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627739.426119:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828553216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627739.426123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627739/real 1778627739] req@ffff95434701c380 x1865013828552832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627739.426148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828552832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627739.426161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627739.426164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627739.426167:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627739.426180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627739.426194:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627739.426217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627739/real 1778627739] req@ffff95434701d500 x1865013828552960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627739.426230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828552960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627739.426239:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627739.426242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627739.426245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828552960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627739.426252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627739.426271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627739.426282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627739/real 1778627739] req@ffff95434701f800 x1865013828553088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627739.426293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828553088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627739.426301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627739.426304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627739.426313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627739.426331:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627739.426350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627739.426360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627739/real 1778627739] req@ffff95434701c700 x1865013828553216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627739.426372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828553216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627739.426380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627739.426383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627739.426386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553216:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778627739.617922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778627739.618090:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627740.385787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627740.641777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627740.641932:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627741.409741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627741.665743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627741.665869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627742.433832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627742.689796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627742.689890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627743.457756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627743.713748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627743.713834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627744.481761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627744.545765:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627744.545773:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627744.545777:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627744.545785:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12079 00000100:00080000:3.0:1778627744.545795:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627744.545800:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627744.545831:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828553344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627744.545859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627744.545862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627744.545863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627744.545866:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12079 00000100:00080000:3.0:1778627744.545868:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627744.545871:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627744.545879:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627744.545888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828553472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627744.545897:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828553344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627744.545898:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627744.545902:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627744.545905:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627744.545910:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12079 00000100:00080000:3.0:1778627744.545912:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627744.545927:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627744.545931:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627744.545940:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828553600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627744.545946:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627744.545949:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627744.545950:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627744.545954:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12079 00000800:00000100:0.0:1778627744.545959:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627744.545961:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627744.545965:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627744.545968:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627744.545977:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828553728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627744.545984:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627744.545987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627744.545990:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627744/real 1778627744] req@ffff95434701c000 x1865013828553344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778627744.545990:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627744.545994:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12079 00000100:00080000:3.0:1778627744.545999:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627744.546004:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627744.546007:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627744.546012:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828553344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627744.546016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828553856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627744.546040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627744.546045:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627744.546055:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828553344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627744.546074:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627744.546091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627744.546114:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627744/real 1778627744] req@ffff95434701ea00 x1865013828553472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627744.546124:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828553472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627744.546130:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627744.546132:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627744.546134:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627744.546139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627744.546153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627744.546159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627744/real 1778627744] req@ffff95434701d880 x1865013828553600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627744.546167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828553600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627744.546172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627744.546173:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627744.546175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627744.546179:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627744.546188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627744.546201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627744/real 1778627744] req@ffff95434701ca80 x1865013828553728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627744.546208:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828553728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627744.546214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627744.546216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627744.546218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627744.546221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627744.546231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627744.546236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627744/real 1778627744] req@ffff95434701d180 x1865013828553856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627744.546243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828553856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627744.546248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627744.546250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627744.546251:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828553856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627744.737785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627744.737847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627745.505767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627745.761777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627745.761853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627746.529794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627746.785823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627746.785933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627747.553738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627747.809791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627747.809829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627748.577862:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627748.833805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627748.833870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627749.601849:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627749.665825:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627749.665838:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627749.665865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627749.665875:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12084 00000100:00080000:0.0:1778627749.665889:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627749.665895:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627749.665934:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828553984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627749.665966:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627749.665969:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627749.665971:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627749.665982:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12084 00000100:00080000:0.0:1778627749.665985:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627749.665989:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627749.666009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627749.666023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828554112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627749.666032:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627749.666036:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627749.666039:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627749.666041:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828553984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627749.666045:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12084 00000100:00080000:0.0:1778627749.666055:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627749.666060:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627749.666064:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627749.666074:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828554240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627749.666098:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627749.666101:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627749.666102:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627749.666106:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12084 00000100:00080000:0.0:1778627749.666114:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627749.666116:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627749.666120:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627749.666127:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627749.666133:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828554368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627749.666141:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627749.666144:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627749.666146:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627749.666150:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12084 00000100:00080000:0.0:1778627749.666153:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627749.666156:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627749.666160:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627749.666170:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828554496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627749.666184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627749/real 1778627749] req@ffff954342f4ad80 x1865013828553984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627749.666225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828553984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627749.666238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627749.666243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627749.666248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828553984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627749.666275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627749.666296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627749.666305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627749/real 1778627749] req@ffff954342f49500 x1865013828554112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627749.666314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828554112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627749.666320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627749.666323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627749.666325:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627749.666329:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627749.666339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627749.666345:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627749/real 1778627749] req@ffff954342f49c00 x1865013828554240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627749.666352:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828554240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627749.666367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627749.666374:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627749.666377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627749.666381:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627749.666396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627749.666403:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627749/real 1778627749] req@ffff954342f4b480 x1865013828554368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627749.666410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828554368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627749.666422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627749.666424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627749.666426:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627749.666431:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627749.666442:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627749.666448:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627749/real 1778627749] req@ffff954342f4a300 x1865013828554496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627749.666455:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828554496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627749.666461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627749.666463:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627749.666469:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627749.857752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627749.857845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627750.625740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627750.881739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627750.881787:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627751.649789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627751.905744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627751.905813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627752.673780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627752.929811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627752.929880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627753.697774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627753.953781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627753.953877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627754.721785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627754.785738:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627754.785745:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627754.785749:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627754.785757:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12089 00000100:00080000:0.0:1778627754.785766:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627754.785772:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627754.785802:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828554624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627754.785823:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627754.785826:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627754.785827:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627754.785830:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12089 00000100:00080000:0.0:1778627754.785833:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627754.785835:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627754.785843:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627754.785850:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828554752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627754.785857:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627754.785860:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627754.785861:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627754.785863:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12089 00000100:00080000:0.0:1778627754.785865:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627754.785867:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627754.785871:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627754.785888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828554880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627754.785908:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627754.785912:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627754.785913:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627754.785920:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12089 00000100:00080000:0.0:1778627754.785925:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627754.785929:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627754.785929:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828554624:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627754.785936:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627754.785952:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828555008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627754.785958:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627754.785961:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627754.785962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627754.785964:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12089 00000100:00080000:0.0:1778627754.785970:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627754.785972:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627754.785975:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627754.785981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828555136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627754.786035:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627754.786081:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627754/real 1778627754] req@ffff954342f49880 x1865013828554624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627754.786102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828554624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627754.786115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627754.786120:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627754.786124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828554624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627754.786133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627754.786157:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627754.786170:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627754/real 1778627754] req@ffff954342f4bb80 x1865013828554752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627754.786183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828554752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627754.786192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627754.786196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627754.786212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627754.786244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627754.786269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627754.786281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627754/real 1778627754] req@ffff954342f49180 x1865013828554880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627754.786295:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828554880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627754.786304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627754.786307:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627754.786311:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828554880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627754.786319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627754.786341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627754.786352:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627754/real 1778627754] req@ffff954342f48000 x1865013828555008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627754.786379:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828555008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627754.786389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627754.786393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627754.786396:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627754.786403:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627754.786425:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627754.786446:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627754/real 1778627754] req@ffff954342f48380 x1865013828555136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627754.786459:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828555136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627754.786468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627754.786472:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627754.786475:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627754.977761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627754.977848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627755.745817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627756.001779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627756.001887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627756.769805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627757.025767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627757.025873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627757.793862:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627758.049820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627758.049910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627758.817798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627759.073777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627759.073847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627759.841855:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627759.905776:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627759.905789:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627759.905811:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627759.905825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12094 00000100:00080000:0.0:1778627759.905841:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627759.905849:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627759.905901:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828555264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627759.905933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627759.905937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627759.905938:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627759.905943:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12094 00000100:00080000:0.0:1778627759.905947:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627759.905951:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627759.905963:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627759.905979:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828555264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627759.905980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828555392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627759.905999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627759.906003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627759.906005:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627759.906010:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12094 00000100:00080000:0.0:1778627759.906015:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627759.906019:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627759.906044:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627759.906060:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828555520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627759.906071:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627759.906078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627759.906083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627759.906085:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627759.906091:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12094 00000100:00080000:0.0:1778627759.906107:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627759.906113:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627759.906120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627759.906123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627759/real 1778627759] req@ffff954342f4aa00 x1865013828555264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627759.906137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828555648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627759.906149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627759.906150:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828555264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627759.906161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627759.906166:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627759.906174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627759.906175:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12094 00000100:00080000:0.0:1778627759.906183:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627759.906184:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778627759.906193:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627759.906194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828555264:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627759.906204:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627759.906220:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828555776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627759.906229:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627759.906259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627759.906283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627759/real 1778627759] req@ffff954342f48e00 x1865013828555392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627759.906296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828555392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627759.906305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627759.906309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627759.906312:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627759.906318:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627759.906338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627759.906348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627759/real 1778627759] req@ffff954342f48380 x1865013828555520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627759.906360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828555520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627759.906368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627759.906371:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627759.906375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627759.906381:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627759.906399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627759.906408:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627759/real 1778627759] req@ffff954342f48000 x1865013828555648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627759.906419:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828555648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627759.906433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627759.906437:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627759.906440:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627759.906456:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627759.906477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627759.906508:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627759/real 1778627759] req@ffff954342f49180 x1865013828555776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627759.906522:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828555776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627759.906531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627759.906534:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627759.906538:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828555776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627760.097826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627760.097916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627760.865820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627761.121767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627761.121836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627761.889808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627762.145786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627762.145872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627762.913772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627763.169774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627763.169939:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627763.937774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627764.193783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627764.193857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627764.961740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627765.025761:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627765.025771:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627765.025776:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627765.025786:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12099 00000100:00080000:0.0:1778627765.025797:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627765.025805:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627765.025835:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828555904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627765.025859:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627765.025868:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627765.025869:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627765.025872:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12099 00000100:00080000:0.0:1778627765.025875:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627765.025878:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627765.025886:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627765.025895:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828556032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627765.025902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627765.025905:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627765.025906:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627765.025909:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12099 00000100:00080000:0.0:1778627765.025911:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627765.025915:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627765.025915:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828555904:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627765.025933:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627765.025946:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828556160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627765.025958:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627765.025962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627765.025964:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627765.025968:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12099 00000100:00080000:0.0:1778627765.025974:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627765.025978:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627765.025984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627765.025985:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627765.025999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828556288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627765.026007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627765.026017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627765.026018:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627765.026022:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12099 00000100:00080000:0.0:1778627765.026025:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627765.026027:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627765/real 1778627765] req@ffff954342f4bb80 x1865013828555904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627765.026030:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627765.026036:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627765.026047:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828556416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627765.026058:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828555904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627765.026084:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627765.026089:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627765.026094:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828555904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627765.026103:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627765.026131:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627765.026145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627765/real 1778627765] req@ffff954342f49880 x1865013828556032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627765.026159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828556032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627765.026169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627765.026173:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627765.026177:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627765.026204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627765.026224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627765.026235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627765/real 1778627765] req@ffff954342f4a300 x1865013828556160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627765.026249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828556160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627765.026257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627765.026260:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627765.026264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627765.026278:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627765.026303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627765.026316:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627765/real 1778627765] req@ffff954342f4b480 x1865013828556288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627765.026331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828556288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627765.026340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627765.026343:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627765.026347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627765.026354:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627765.026375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627765.026386:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627765/real 1778627765] req@ffff954342f49c00 x1865013828556416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627765.026398:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828556416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627765.026407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627765.026411:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627765.026414:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627765.217787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627765.217841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627765.985787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627766.241763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627766.241843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627767.009733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627767.265747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627767.265818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627768.033777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627768.289772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627768.289828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627769.057741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627769.313757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627769.313809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627770.081764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627770.145766:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627770.145774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627770.145778:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627770.145792:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12104 00000100:00080000:0.0:1778627770.145804:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627770.145810:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627770.145843:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828556544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627770.145866:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627770.145868:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627770.145870:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627770.145873:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12104 00000100:00080000:0.0:1778627770.145875:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627770.145877:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627770.145884:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627770.145893:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828556672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627770.145905:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627770.145907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627770.145908:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627770.145911:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12104 00000100:00080000:0.0:1778627770.145912:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627770.145915:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627770.145918:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627770.145940:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828556800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627770.145947:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828556544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627770.145953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627770.145962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627770.145965:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627770.145970:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12104 00000100:00080000:0.0:1778627770.145975:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627770.145980:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627770.145986:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627770.146007:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828556928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627770.146014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627770.146017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627770.146018:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627770.146021:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12104 00000100:00080000:0.0:1778627770.146024:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627770.146027:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627770.146031:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627770.146040:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828557056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627770.146043:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627770.146077:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627770.146095:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627770.146109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627770.337748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627770.337814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627771.105796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627771.361754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627771.361826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627772.129762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627772.385741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627772.385832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627773.153779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778627773.217938:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778627773.217991:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778627773.218008:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627773.218053:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627773.218063:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627773.218074:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778627773.218082:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778627773.218157:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627770/real 1778627773] req@ffff954342f48700 x1865013828556800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627773.218189:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828556800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627773.218202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627773.218208:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627773.218213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627773.218222:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627770/real 1778627773] req@ffff954342f4b100 x1865013828556928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627773.218235:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828556928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627773.218243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627773.218246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627773.218249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627773.218256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627770/real 1778627773] req@ffff954342f49f80 x1865013828557056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627773.218293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828557056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627773.218302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627773.218306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627773.218310:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627773.218316:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627770/real 1778627773] req@ffff954342f49500 x1865013828556544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627773.218329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828556544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627773.218337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627773.218340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627773.218344:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828556544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627773.218349:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627770/real 1778627773] req@ffff954342f4ad80 x1865013828556672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627773.218361:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828556672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627773.218369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627773.218372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627773.218376:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828556672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778627773.409766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627773.409848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627774.177768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627774.433752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627774.433816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627775.201831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627775.265812:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627775.265821:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627775.265825:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627775.265832:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12109 00000100:00080000:1.0:1778627775.265842:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627775.265848:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627775.265877:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828557184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627775.265916:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627775.265920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627775.265922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627775.265928:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12109 00000100:00080000:1.0:1778627775.265933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627775.265939:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627775.265949:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627775.265959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828557184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627775.265967:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828557312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627775.265986:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627775.265991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627775.265993:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627775.265997:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12109 00000100:00080000:1.0:1778627775.266002:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627775.266006:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627775.266014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627775.266027:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828557440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627775.266037:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627775.266041:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627775.266043:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627775.266047:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12109 00000800:00000100:2.0:1778627775.266052:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627775.266062:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627775.266068:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627775.266084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627775.266089:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627775/real 1778627775] req@ffff954350197100 x1865013828557184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627775.266101:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828557568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627775.266114:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627775.266117:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828557184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627775.266119:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627775.266122:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627775.266132:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12109 00000100:00080000:1.0:1778627775.266139:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627775.266139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627775.266146:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778627775.266149:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:1.0:1778627775.266154:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627775.266159:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828557184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627775.266169:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828557696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627775.266190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627775.266220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627775.266244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627775/real 1778627775] req@ffff954350197800 x1865013828557312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627775.266257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828557312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627775.266265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627775.266268:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627775.266271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627775.266277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627775.266299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627775.266310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627775/real 1778627775] req@ffff954350196d80 x1865013828557440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627775.266338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828557440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627775.266345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627775.266349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627775.266352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627775.266359:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627775.266385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627775.266398:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627775/real 1778627775] req@ffff954350197480 x1865013828557568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627775.266410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828557568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627775.266418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627775.266421:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627775.266424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627775.266431:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627775.266452:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627775.266463:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627775/real 1778627775] req@ffff954350194700 x1865013828557696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627775.266474:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828557696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627775.266511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627775.266515:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627775.266519:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627775.457777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627775.457834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627776.225760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627776.481760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627776.481849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627777.249780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627777.505763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627777.505825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627778.273806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627778.529747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627778.529894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627779.297798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627779.553775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627779.553839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627780.321776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627780.385767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627780.385776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627780.385780:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627780.385788:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12114 00000100:00080000:2.0:1778627780.385798:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627780.385804:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627780.385836:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828557824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627780.385862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627780.385865:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627780.385866:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627780.385869:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12114 00000100:00080000:2.0:1778627780.385872:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627780.385874:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627780.385881:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627780.385891:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828557952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627780.385899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627780.385903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828557824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627780.385903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627780.385926:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627780.385930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12114 00000100:00080000:2.0:1778627780.385934:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627780.385937:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627780.385941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627780.385952:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828558080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627780.385959:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627780.385965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627780.385968:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627780.385970:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627780.385974:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12114 00000100:00080000:2.0:1778627780.385978:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627780.385980:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627780.385984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627780.385996:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627780/real 1778627780] req@ffff95435878a680 x1865013828557824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627780.385997:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828558208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627780.386022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627780.386024:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828557824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627780.386031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627780.386036:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627780.386038:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627780.386044:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12114 00000100:00080000:0.0:1778627780.386045:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627780.386052:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828557824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627780.386052:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627780.386060:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627780.386065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627780.386075:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627780.386083:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828558336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627780.386094:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627780.386105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627780/real 1778627780] req@ffff954358789f80 x1865013828557952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627780.386115:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828557952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627780.386121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627780.386124:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627780.386126:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828557952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627780.386137:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627780.386151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627780.386157:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627780/real 1778627780] req@ffff95435878ad80 x1865013828558080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627780.386165:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828558080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627780.386170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627780.386172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627780.386173:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627780.386177:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627780.386187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627780.386193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627780/real 1778627780] req@ffff95435878a300 x1865013828558208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627780.386200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828558208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627780.386205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627780.386206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627780.386208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627780.386220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627780.386236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627780.386246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627780/real 1778627780] req@ffff95435878b800 x1865013828558336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627780.386266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828558336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627780.386275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627780.386279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627780.386282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627780.577745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627780.577816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627781.345783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627781.601785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627781.601846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627782.369757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627782.625781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627782.625844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627783.393784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627783.649772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627783.649841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627784.417767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627784.673765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627784.673831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627785.441767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627785.505783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627785.505791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627785.505796:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627785.505811:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12120 00000100:00080000:2.0:1778627785.505821:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627785.505827:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627785.505861:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828558464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627785.505884:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627785.505887:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627785.505888:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627785.505891:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12120 00000100:00080000:2.0:1778627785.505894:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627785.505896:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627785.505903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627785.505911:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828558592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627785.505918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627785.505921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627785.505922:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627785.505925:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12120 00000100:00080000:2.0:1778627785.505927:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627785.505934:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828558464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627785.505945:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627785.505948:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627785.505959:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828558720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627785.505966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627785.505969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627785.505970:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627785.505973:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12120 00000100:00080000:2.0:1778627785.505976:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627785.505979:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627785.505982:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627785.505989:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627785.505992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828558848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627785.505999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627785.506001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627785.506003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627785.506005:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12120 00000100:00080000:2.0:1778627785.506008:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627785.506010:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627785.506013:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627785.506020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828558976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627785.506027:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627785/real 1778627785] req@ffff954358788a80 x1865013828558464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627785.506043:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828558464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627785.506051:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627785.506055:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627785.506058:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828558464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627785.506063:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627785.506080:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627785.506092:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627785/real 1778627785] req@ffff954358789180 x1865013828558592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627785.506115:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828558592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627785.506125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627785.506129:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627785.506132:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627785.506152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627785.506168:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627785.506175:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627785/real 1778627785] req@ffff954358788e00 x1865013828558720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627785.506183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828558720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627785.506188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627785.506190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627785.506192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627785.506195:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627785.506205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627785.506211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627785/real 1778627785] req@ffff954358789c00 x1865013828558848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627785.506217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828558848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627785.506238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627785.506242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627785.506245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627785.506251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627785.506273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627785.506283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627785/real 1778627785] req@ffff954358789880 x1865013828558976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627785.506307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828558976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627785.506313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627785.506315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627785.506317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828558976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627785.697776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627785.697836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627786.465761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627786.721757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627786.721826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627787.489807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627787.745791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627787.745840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627788.513799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627788.769744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627788.769817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627789.537745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627789.793771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627789.793837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627790.561773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627790.625802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627790.625809:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627790.625813:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627790.625821:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12125 00000100:00080000:0.0:1778627790.625832:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627790.625840:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627790.625893:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828559104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627790.625917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627790.625920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627790.625921:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627790.625925:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12125 00000100:00080000:0.0:1778627790.625928:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627790.625932:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627790.625939:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627790.625949:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828559232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627790.625957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627790.625959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828559104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627790.625960:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627790.625962:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627790.625965:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12125 00000100:00080000:0.0:1778627790.625967:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627790.625971:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627790.625978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627790.625987:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828559360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627790.625992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627790.625994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627790.625996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627790.626004:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12125 00000100:00080000:0.0:1778627790.626008:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627790.626012:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627790.626016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627790.626024:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828559488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627790.626031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627790.626033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627790.626034:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627790.626036:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12125 00000100:00080000:0.0:1778627790.626039:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627790.626043:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627790.626044:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627790.626049:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627790.626056:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828559616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627790.626083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627790/real 1778627790] req@ffff954342f4a680 x1865013828559104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627790.626104:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828559104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627790.626115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627790.626119:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627790.626123:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828559104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627790.626146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627790.626175:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627790.626200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627790/real 1778627790] req@ffff954342f48a80 x1865013828559232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627790.626214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828559232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627790.626224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627790.626227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627790.626231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627790.626238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627790.626256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627790.626268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627790/real 1778627790] req@ffff954342f4ad80 x1865013828559360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627790.626293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828559360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627790.626302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627790.626305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627790.626309:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627790.626315:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627790.626335:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627790.626346:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627790/real 1778627790] req@ffff954342f49500 x1865013828559488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627790.626366:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828559488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627790.626375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627790.626378:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627790.626381:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627790.626388:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627790.626408:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627790.626419:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627790/real 1778627790] req@ffff954342f49f80 x1865013828559616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627790.626431:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828559616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627790.626452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627790.626456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627790.626459:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627790.817739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627790.817810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627791.585781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627791.841778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627791.841858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627792.609837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627792.865819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627792.865893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627793.633770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627793.889793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627793.889864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627794.657832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627794.913813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627794.913898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627795.681814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627795.745820:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627795.745832:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627795.745839:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627795.745852:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12130 00000100:00080000:0.0:1778627795.745868:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627795.745904:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627795.745946:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828559744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627795.745982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627795.745985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627795.745987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627795.745990:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12130 00000100:00080000:0.0:1778627795.745994:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627795.745997:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627795.746006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627795.746023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828559872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627795.746031:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627795.746034:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627795.746035:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627795.746041:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12130 00000100:00080000:0.0:1778627795.746045:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627795.746045:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828559744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627795.746049:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627795.746056:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627795.746064:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828560000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627795.746069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627795.746072:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627795.746079:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627795.746083:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12130 00000100:00080000:0.0:1778627795.746103:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627795.746108:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627795.746113:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627795.746123:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828560128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627795.746131:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627795.746134:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627795.746135:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627795.746138:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12130 00000100:00080000:0.0:1778627795.746141:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627795.746143:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627795.746147:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627795.746150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627795.746158:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828560256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627795.746205:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627795/real 1778627795] req@ffff954342f4b100 x1865013828559744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627795.746231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828559744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627795.746242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627795.746246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627795.746249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828559744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627795.746275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627795.746294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627795.746304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627795/real 1778627795] req@ffff954342f48700 x1865013828559872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627795.746312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828559872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627795.746319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627795.746321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627795.746323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828559872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627795.746335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627795.746349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627795.746356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627795/real 1778627795] req@ffff954342f49c00 x1865013828560000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627795.746364:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828560000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627795.746370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627795.746372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627795.746374:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627795.746378:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627795.746389:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627795.746394:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627795/real 1778627795] req@ffff954342f4b480 x1865013828560128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627795.746402:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828560128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627795.746408:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627795.746410:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627795.746411:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627795.746415:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627795.746424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627795.746429:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627795/real 1778627795] req@ffff954342f4a300 x1865013828560256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627795.746443:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828560256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627795.746449:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627795.746452:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627795.746454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627795.937779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627795.937852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627796.705788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627796.961827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627796.961913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627797.729793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627797.985775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627797.985833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627798.753777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627799.009735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627799.009800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627799.777768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627800.033754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627800.033810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627800.801772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627800.865812:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627800.865823:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627800.865828:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627800.865836:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12135 00000100:00080000:1.0:1778627800.865852:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627800.865858:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627800.865891:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828560384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627800.865911:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627800.865914:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627800.865916:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627800.865919:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12135 00000100:00080000:1.0:1778627800.865921:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627800.865923:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627800.865930:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627800.865939:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828560512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627800.865947:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627800.865962:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828560384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627800.865964:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627800.865969:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627800.865975:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12135 00000100:00080000:1.0:1778627800.865979:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627800.865984:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627800.866000:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627800.866017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828560640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627800.866029:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627800.866033:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627800.866035:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627800.866040:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12135 00000100:00080000:1.0:1778627800.866044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627800.866046:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627800.866049:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627800.866055:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627800.866072:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828560768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627800.866079:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627800.866084:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778627800.866085:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627800/real 1778627800] req@ffff954350194000 x1865013828560384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627800.866086:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627800.866096:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12135 00000100:00080000:1.0:1778627800.866103:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627800.866109:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627800.866114:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828560384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627800.866124:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627800.866130:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627800.866136:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627800.866141:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828560896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627800.866144:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828560384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627800.866161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627800.866184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627800.866195:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627800/real 1778627800] req@ffff954350194a80 x1865013828560512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627800.866215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828560512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627800.866225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627800.866228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627800.866232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627800.866255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627800.866272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627800.866281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627800/real 1778627800] req@ffff954350194380 x1865013828560640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627800.866293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828560640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627800.866301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627800.866304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627800.866307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627800.866313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627800.866328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627800.866336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627800/real 1778627800] req@ffff954350194e00 x1865013828560768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627800.866347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828560768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627800.866367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627800.866370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627800.866374:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627800.866380:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627800.866397:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627800.866405:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627800/real 1778627800] req@ffff954350195500 x1865013828560896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627800.866417:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828560896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627800.866426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627800.866429:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627800.866432:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828560896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778627801.057750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627801.057805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627801.825751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778627802.082397:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627802.082863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627802.849794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778627803.106001:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627803.106046:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627803.873744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778627804.129959:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627804.130146:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627804.897842:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778627805.154614:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627805.154695:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627805.921740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627805.985765:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627805.985775:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627805.985780:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627805.985791:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12140 00000100:00080000:1.0:1778627805.985804:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627805.985810:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627805.985858:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828561024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627805.985897:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627805.985901:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627805.985904:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627805.985908:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12140 00000100:00080000:1.0:1778627805.985912:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627805.985917:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627805.985928:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627805.985942:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828561152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627805.985942:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828561024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627805.985952:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627805.985956:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627805.985992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627805.985996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12140 00000100:00080000:1.0:1778627805.986000:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627805.986004:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627805.986009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627805.986020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828561280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627805.986030:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778627805.986031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627805.986033:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627805.986035:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627805.986039:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12140 00000100:00080000:1.0:1778627805.986043:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627805.986046:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627805.986050:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627805.986059:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828561408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627805.986067:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627805.986070:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627805.986071:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627805.986075:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12140 00000100:00080000:1.0:1778627805.986078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627805.986078:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627805/real 1778627805] req@ffff954350196a00 x1865013828561024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627805.986082:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627805.986086:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627805.986100:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828561024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627805.986108:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828561536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627805.986117:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627805.986122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627805.986127:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828561024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627805.986157:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627805.986180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627805.986189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627805/real 1778627805] req@ffff954350195880 x1865013828561152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627805.986198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828561152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627805.986205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627805.986207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627805.986209:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627805.986213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627805.986224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627805.986229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627805/real 1778627805] req@ffff954350197b80 x1865013828561280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627805.986237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828561280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627805.986242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627805.986244:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627805.986245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627805.986255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627805.986270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627805.986276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627805/real 1778627805] req@ffff954350195180 x1865013828561408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627805.986283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828561408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627805.986289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627805.986290:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627805.986292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627805.986295:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627805.986305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627805.986310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627805/real 1778627805] req@ffff954350195c00 x1865013828561536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627805.986317:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828561536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627805.986323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627805.986325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627805.986327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627806.178020:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627806.178125:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627806.945819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627807.201831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627807.202107:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627807.969790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627808.225788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627808.225885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627808.993790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627809.249805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627809.249878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627810.017774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627810.273762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627810.273865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627811.041806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627811.105776:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627811.105793:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627811.105798:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627811.105807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12145 00000100:00080000:0.0:1778627811.105817:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627811.105823:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627811.105855:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828561664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627811.105878:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627811.105881:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627811.105882:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627811.105885:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12145 00000100:00080000:0.0:1778627811.105887:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627811.105890:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627811.105896:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627811.105904:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828561792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627811.105911:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627811.105914:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627811.105930:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627811.105933:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12145 00000100:00080000:0.0:1778627811.105936:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627811.105940:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627811.105944:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627811.105956:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828561920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627811.105959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828561664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627811.105965:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627811.105967:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627811.105968:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627811.105972:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12145 00000100:00080000:0.0:1778627811.105976:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627811.105979:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627811.105989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627811.105996:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828562048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627811.106001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627811.106003:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627811.106004:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627811.106007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12145 00000100:00080000:0.0:1778627811.106009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627811.106011:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627811.106013:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627811.106028:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828562176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627811.106080:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627811.106123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627811/real 1778627811] req@ffff954342f49880 x1865013828561664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627811.106142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828561664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627811.106151:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627811.106155:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627811.106158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828561664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627811.106164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627811.106179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627811.106187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627811/real 1778627811] req@ffff954342f4bb80 x1865013828561792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627811.106194:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828561792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627811.106200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627811.106202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627811.106204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627811.106228:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627811.106238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627811.106244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627811/real 1778627811] req@ffff954342f49180 x1865013828561920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627811.106265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828561920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627811.106271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627811.106273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627811.106275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828561920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627811.106279:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627811.106292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627811.106298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627811/real 1778627811] req@ffff954342f48000 x1865013828562048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627811.106306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828562048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627811.106311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627811.106313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627811.106314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627811.106318:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627811.106328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627811.106333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627811/real 1778627811] req@ffff954342f48380 x1865013828562176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627811.106340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828562176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627811.106355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627811.106357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627811.106359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627811.297767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627811.297879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627812.065820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627812.321844:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627812.321912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627813.089778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627813.345801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627813.345903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627814.113826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627814.369812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627814.369946:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627815.137796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627815.393791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627815.393857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627816.161844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627816.225803:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627816.225817:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627816.225824:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627816.225836:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12150 00000100:00080000:1.0:1778627816.225851:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627816.225859:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627816.225910:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828562304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627816.225943:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627816.225947:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627816.225949:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627816.225955:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12150 00000100:00080000:1.0:1778627816.225960:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627816.225965:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627816.225981:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828562304:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778627816.226000:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627816.226020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828562432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627816.226033:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627816.226037:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627816.226039:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627816.226044:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12150 00000100:00080000:1.0:1778627816.226049:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627816.226053:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627816.226059:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627816.226062:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627816.226078:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828562560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627816.226089:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627816.226092:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627816.226095:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627816.226100:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12150 00000100:00080000:1.0:1778627816.226105:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627816.226110:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627816/real 1778627816] req@ffff954350195f80 x1865013828562304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627816.226114:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627816.226123:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627816.226138:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828562688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627816.226148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828562304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627816.226151:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627816.226164:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627816.226166:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627816.226172:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627816.226173:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627816.226181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828562304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627816.226183:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12150 00000100:00080000:1.0:1778627816.226191:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627816.226197:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627816.226203:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627816.226216:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828562816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627816.226219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627816.226258:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627816.226273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627816/real 1778627816] req@ffff954350195c00 x1865013828562432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627816.226287:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828562432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627816.226305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627816.226309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627816.226313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627816.226320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627816.226344:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627816.226357:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627816/real 1778627816] req@ffff954350195180 x1865013828562560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627816.226370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828562560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627816.226378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627816.226381:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627816.226385:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627816.226391:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627816.226415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627816.226427:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627816/real 1778627816] req@ffff954350197b80 x1865013828562688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627816.226440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828562688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627816.226450:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627816.226453:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627816.226463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627816.226481:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627816.226504:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627816.226516:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627816/real 1778627816] req@ffff954350195880 x1865013828562816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627816.226529:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828562816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627816.226539:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627816.226543:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627816.226546:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828562816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627816.417775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627816.417844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627817.185818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627817.441739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627817.441834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627818.209835:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627818.465819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627818.465899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627819.233817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627819.489809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627819.489880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627820.257818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627820.513820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627820.513887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627821.281818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627821.345797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627821.345807:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627821.345812:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627821.345822:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12155 00000100:00080000:2.0:1778627821.345835:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627821.345841:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627821.345883:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828562944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627821.345913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627821.345930:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627821.345932:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627821.345937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12155 00000100:00080000:2.0:1778627821.345941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627821.345944:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627821.345954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627821.345960:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828562944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627821.345966:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828563072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627821.345976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627821.345978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627821.345980:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627821.345983:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12155 00000100:00080000:2.0:1778627821.345985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627821.345987:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627821.345990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627821.345999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828563200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627821.346005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627821.346008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627821.346009:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627821.346012:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12155 00000100:00080000:2.0:1778627821.346014:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627821.346017:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627821.346024:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627821.346037:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627821.346037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828563328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627821.346047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627821.346049:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627821.346051:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627821.346054:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12155 00000100:00080000:2.0:1778627821.346058:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627821.346060:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627821.346063:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627821.346070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828563456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627821.346078:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627821/real 1778627821] req@ffff954358789500 x1865013828562944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627821.346097:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828562944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627821.346106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627821.346110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627821.346113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828562944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627821.346136:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627821.346151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627821.346159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627821/real 1778627821] req@ffff954358788700 x1865013828563072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627821.346179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828563072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627821.346186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627821.346188:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627821.346190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627821.346194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627821.346208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627821.346214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627821/real 1778627821] req@ffff95435878b100 x1865013828563200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627821.346222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828563200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627821.346226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627821.346228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627821.346230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627821.346233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627821.346244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627821.346250:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627821/real 1778627821] req@ffff95435878bb80 x1865013828563328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627821.346257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828563328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627821.346271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627821.346274:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627821.346276:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627821.346279:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627821.346292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627821.346298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627821/real 1778627821] req@ffff95435878b480 x1865013828563456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627821.346305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828563456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627821.346311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627821.346312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627821.346314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627821.537788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627821.537861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627822.305796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627822.561810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627822.561874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627823.329805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627823.585753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627823.585809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627824.353796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627824.609823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627824.609896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627825.377758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627825.633752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627825.633846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627826.401831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627826.465808:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627826.465823:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627826.465829:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627826.465843:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12160 00000100:00080000:2.0:1778627826.465858:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627826.465867:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627826.465917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828563584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627826.465978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627826.465983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627826.465986:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627826.465993:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12160 00000100:00080000:2.0:1778627826.465999:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627826.466005:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627826.466018:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627826.466023:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828563584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627826.466038:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828563712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627826.466053:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627826.466057:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627826.466060:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627826.466064:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12160 00000100:00080000:2.0:1778627826.466069:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627826.466074:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627826.466080:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627826.466108:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828563840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627826.466114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627826.466133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627826.466137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627826.466140:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627826.466148:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12160 00000100:00100000:0.0:1778627826.466158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627826/real 1778627826] req@ffff95435878aa00 x1865013828563584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627826.466163:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627826.466174:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627826.466180:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828563584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778627826.466183:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627826.466206:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828563968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627826.466209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627826.466215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627826.466218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828563584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627826.466242:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627826.466246:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627826.466248:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627826.466253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563712:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778627826.466268:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12160 00000100:00080000:2.0:1778627826.466282:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627826.466289:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627826.466298:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627826.466307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627826.466322:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828564096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627826.466324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627826/real 1778627826] req@ffff954358788380 x1865013828563712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627826.466355:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828563712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627826.466397:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627826.466406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627826.466411:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627826.466422:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627826.466444:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627826.466470:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627826/real 1778627826] req@ffff95435878b480 x1865013828563840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627826.466502:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828563840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627826.466510:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627826.466513:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627826.466517:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627826.466549:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627826.466581:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627826.466593:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627826/real 1778627826] req@ffff95435878bb80 x1865013828563968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627826.466607:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828563968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627826.466614:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627826.466618:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627826.466621:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828563968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627826.466639:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627826.466659:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627826.466689:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627826/real 1778627826] req@ffff95435878b100 x1865013828564096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627826.466699:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828564096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627826.466705:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627826.466707:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627826.466710:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627826.657811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627826.657890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627827.425817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627827.681745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627827.681856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627828.449796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627828.705801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627828.705905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627829.473865:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627829.729808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627829.729907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627830.497806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627830.753793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627830.753881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627831.521838:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627831.585781:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627831.585792:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627831.585796:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627831.585807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12166 00000100:00080000:1.0:1778627831.585819:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627831.585826:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627831.585869:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828564224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627831.585914:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627831.585917:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627831.585918:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627831.585922:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12166 00000100:00080000:1.0:1778627831.585925:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627831.585928:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627831.585938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627831.585948:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828564352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627831.585956:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627831.585960:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828564224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627831.585960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627831.585962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627831.585969:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12166 00000100:00080000:1.0:1778627831.585972:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627831.585974:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627831.585977:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627831.585984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828564480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627831.585991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627831.585994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627831.585995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627831.585999:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12166 00000100:00080000:1.0:1778627831.586011:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627831.586017:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627831.586023:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627831.586036:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828564608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627831.586047:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778627831.586051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627831.586053:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627831.586056:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627831.586060:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12166 00000100:00080000:1.0:1778627831.586068:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627831.586072:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627831.586076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627831.586087:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828564736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627831.586106:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627831/real 1778627831] req@ffff954350196a00 x1865013828564224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627831.586125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828564224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627831.586135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627831.586138:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627831.586142:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828564224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627831.586164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627831.586182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627831.586191:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627831/real 1778627831] req@ffff954350195500 x1865013828564352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627831.586199:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828564352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627831.586205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627831.586207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627831.586209:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627831.586212:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627831.586223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627831.586228:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627831/real 1778627831] req@ffff954350194e00 x1865013828564480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627831.586253:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828564480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627831.586258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627831.586261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627831.586263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627831.586266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627831.586282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627831.586288:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627831/real 1778627831] req@ffff954350194380 x1865013828564608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627831.586296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828564608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627831.586301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627831.586303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627831.586304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627831.586307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627831.586317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627831.586322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627831/real 1778627831] req@ffff954350194a80 x1865013828564736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627831.586329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828564736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627831.586344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627831.586346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627831.586348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627831.777805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627831.777884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627832.545848:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627832.801774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627832.801849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627833.569843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627833.825813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627833.825904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627834.593809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627834.849753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627834.849859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627835.617868:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627835.873791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627835.873892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627836.641803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627836.705821:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627836.705834:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627836.705840:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627836.705854:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12171 00000100:00080000:1.0:1778627836.705870:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627836.705878:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627836.705927:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828564864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627836.705963:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627836.705987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627836.705990:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627836.705996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12171 00000100:00080000:1.0:1778627836.706001:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627836.706007:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627836.706019:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627836.706037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828564992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627836.706051:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627836.706055:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828564864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627836.706056:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627836.706059:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627836.706071:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12171 00000100:00080000:1.0:1778627836.706075:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627836.706080:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627836.706085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627836.706099:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828565120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627836.706109:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627836.706113:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627836.706115:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627836.706119:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12171 00000100:00080000:1.0:1778627836.706133:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627836.706138:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627836.706153:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627836.706169:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828565248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627836.706180:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627836.706185:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627836.706189:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627836.706190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564992:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627836.706201:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12171 00000100:00080000:1.0:1778627836.706208:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627836.706215:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627836.706222:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627836.706240:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828565376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627836.706243:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627836.706312:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627836.706344:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627836.897818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627836.897892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627837.665782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627837.921787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627837.921845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627838.689761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627838.945760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627838.945850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627839.713843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627839.777931:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627839.777968:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627839.777983:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627839.778019:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627839.778026:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627839.778032:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627839.778038:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627839.778074:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627836/real 1778627839] req@ffff954350197800 x1865013828565376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627839.778102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828565376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627839.778115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627839.778121:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627839.778139:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627839.778148:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627836/real 1778627839] req@ffff954350194700 x1865013828564992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627839.778163:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828564992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627839.778179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627839.778183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627839.778187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828564992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627839.778194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627836/real 1778627839] req@ffff954350197480 x1865013828565120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627839.778207:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828565120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627839.778215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627839.778232:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627839.778236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627839.778243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627836/real 1778627839] req@ffff954350196d80 x1865013828565248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627839.778257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828565248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627839.778266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627839.778269:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627839.778283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627839.778290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627836/real 1778627839] req@ffff954350194000 x1865013828564864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627839.778305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828564864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627839.778313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627839.778317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627839.778320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828564864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627839.969781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627839.969842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627840.737773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627840.993782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627840.993850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627841.761798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627841.825774:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627841.825783:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627841.825787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627841.825795:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12176 00000100:00080000:1.0:1778627841.825805:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627841.825811:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627841.825841:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828565504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627841.825871:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627841.825875:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627841.825877:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627841.825882:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12176 00000100:00080000:1.0:1778627841.825903:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627841.825908:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627841.825918:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627841.825930:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828565504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627841.825938:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828565632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627841.825952:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627841.825956:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627841.825964:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627841.825968:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12176 00000100:00080000:1.0:1778627841.825987:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627841.825991:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627841.825997:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778627841.825998:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627841.826015:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828565760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627841.826027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627841.826030:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627841.826032:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627841.826037:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12176 00000100:00080000:1.0:1778627841.826042:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627841.826047:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627841/real 1778627841] req@ffff954350197100 x1865013828565504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627841.826050:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627841.826060:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627841.826078:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828565888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627841.826081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828565504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627841.826115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627841.826117:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627841.826122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627841.826123:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627841.826127:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627841.826132:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828565504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627841.826136:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12176 00000100:00080000:1.0:1778627841.826145:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627841.826155:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627841.826160:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627841.826188:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627841.826188:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828566016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627841.826225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627841.826243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627841/real 1778627841] req@ffff954350194000 x1865013828565632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627841.826264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828565632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627841.826274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627841.826277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627841.826281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627841.826290:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627841.826315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627841.826326:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627841/real 1778627841] req@ffff954350196d80 x1865013828565760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627841.826338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828565760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627841.826346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627841.826349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627841.826353:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627841.826359:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627841.826380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627841.826391:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627841/real 1778627841] req@ffff954350197480 x1865013828565888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627841.826402:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828565888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627841.826424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627841.826428:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627841.826431:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828565888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627841.826450:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627841.826472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627841.826483:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627841/real 1778627841] req@ffff954350194700 x1865013828566016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627841.826496:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828566016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627841.826505:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627841.826508:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627841.826511:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627842.017765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627842.017833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627842.785743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627843.041766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627843.041823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627843.809769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627844.065805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627844.065880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627844.833779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627845.089769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627845.089821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627845.857768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627846.113758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627846.113784:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627846.881786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627846.945744:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627846.945752:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627846.945756:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627846.945764:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12181 00000100:00080000:1.0:1778627846.945774:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627846.945780:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627846.945813:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828566144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627846.945835:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627846.945837:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627846.945839:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627846.945842:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12181 00000100:00080000:1.0:1778627846.945844:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627846.945847:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627846.945868:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627846.945870:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828566144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627846.945879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828566272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627846.945889:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627846.945891:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627846.945893:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627846.945895:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12181 00000100:00080000:1.0:1778627846.945898:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627846.945901:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627846.945904:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627846.945911:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828566400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627846.945917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627846.945919:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627846.945920:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627846.945923:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12181 00000100:00080000:1.0:1778627846.945926:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778627846.945927:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627846.945931:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627846.945935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627846.945942:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828566528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627846.945948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627846.945950:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627846.945957:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627846.945961:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627846/real 1778627846] req@ffff954350197800 x1865013828566144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627846.945962:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12181 00000100:00080000:1.0:1778627846.945965:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627846.945967:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627846.945974:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627846.945989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828566656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627846.945993:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828566144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627846.946012:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627846.946015:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627846.946018:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828566144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627846.946037:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627846.946053:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627846.946061:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627846/real 1778627846] req@ffff954350194a80 x1865013828566272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627846.946069:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828566272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627846.946092:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627846.946095:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627846.946097:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627846.946101:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627846.946115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627846.946123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627846/real 1778627846] req@ffff954350194380 x1865013828566400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627846.946131:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828566400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627846.946145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627846.946147:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627846.946150:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627846.946154:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627846.946167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627846.946172:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627846/real 1778627846] req@ffff954350194e00 x1865013828566528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627846.946179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828566528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627846.946185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627846.946187:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627846.946189:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627846.946193:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627846.946202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627846.946207:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627846/real 1778627846] req@ffff954350195500 x1865013828566656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627846.946214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828566656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627846.946219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627846.946221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627846.946227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627847.137744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627847.137818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627847.905745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627848.161786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627848.161809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627848.929787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627849.185751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627849.185790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627849.953755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627850.209759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627850.209831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627850.977781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627851.233767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627851.233814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627852.001735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627852.065765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627852.065773:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627852.065777:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627852.065785:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12186 00000100:00080000:2.0:1778627852.065795:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627852.065801:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627852.065834:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828566784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627852.065856:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627852.065858:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627852.065860:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627852.065863:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12186 00000100:00080000:2.0:1778627852.065865:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627852.065867:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627852.065887:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627852.065896:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828566912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627852.065904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627852.065907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627852.065910:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627852.065914:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12186 00000100:00080000:2.0:1778627852.065918:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627852.065922:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627852.065927:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627852.065928:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828566784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627852.065941:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828567040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627852.065953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627852.065957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627852.065959:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627852.065963:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12186 00000100:00080000:2.0:1778627852.065967:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627852.065971:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627852.065991:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627852.065999:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627852.066003:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828567168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627852.066011:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627852.066013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627852.066020:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627852.066023:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12186 00000100:00080000:2.0:1778627852.066027:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627852.066030:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627852.066035:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627852.066036:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627852/real 1778627852] req@ffff954358788700 x1865013828566784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627852.066045:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828567296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627852.066058:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828566784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627852.066067:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627852.066071:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627852.066074:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828566784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627852.066080:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627852.066094:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627852.066101:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627852/real 1778627852] req@ffff954358789500 x1865013828566912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627852.066109:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828566912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627852.066123:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627852.066125:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627852.066128:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828566912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627852.066146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627852.066158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627852.066165:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627852/real 1778627852] req@ffff954358789880 x1865013828567040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627852.066172:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828567040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627852.066177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627852.066180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627852.066181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627852.066185:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627852.066194:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627852.066199:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627852/real 1778627852] req@ffff954358789c00 x1865013828567168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627852.066206:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828567168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627852.066211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627852.066213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627852.066219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627852.066223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627852.066234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627852.066240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627852/real 1778627852] req@ffff954358788e00 x1865013828567296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627852.066247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828567296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627852.066253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627852.066254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627852.066256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627852.257741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627852.257816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627853.025775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627853.281792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627853.281823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627854.049756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627854.305760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778627854.305823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627855.073795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627855.329775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627855.329808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627856.097781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627856.353743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778627856.353810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627857.121783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627857.185827:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627857.185835:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627857.185840:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627857.185847:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12191 00000100:00080000:0.0:1778627857.185857:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627857.185862:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627857.185915:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828567424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627857.185962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627857.185966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627857.185968:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627857.185974:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12191 00000100:00080000:0.0:1778627857.185979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627857.185984:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627857.185994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627857.186000:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828567424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627857.186014:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828567552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627857.186026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627857.186030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627857.186032:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627857.186036:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12191 00000100:00080000:0.0:1778627857.186041:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627857.186045:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627857.186050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778627857.186061:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627857.186066:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828567680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627857.186078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627857.186082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627857.186084:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627857.186088:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12191 00000100:00080000:0.0:1778627857.186097:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627857.186103:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627857.186103:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627857/real 1778627857] req@ffff954342f48e00 x1865013828567424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778627857.186111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627857.186123:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828567808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627857.186131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627857.186135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778627857.186135:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828567424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627857.186138:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627857.186146:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12191 00000100:00080000:0.0:1778627857.186151:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627857.186157:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627857.186161:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778627857.186161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627857.186169:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627857.186174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828567424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627857.186190:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828567936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627857.186205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627857.186237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627857.186247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627857/real 1778627857] req@ffff954342f4aa00 x1865013828567552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627857.186256:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828567552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627857.186262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627857.186264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627857.186266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627857.186270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627857.186281:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627857.186287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627857/real 1778627857] req@ffff954342f48380 x1865013828567680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627857.186294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828567680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627857.186299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627857.186301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627857.186304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627857.186310:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627857.186320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627857.186325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627857/real 1778627857] req@ffff954342f48000 x1865013828567808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627857.186342:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828567808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627857.186347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627857.186349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627857.186351:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627857.186355:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627857.186370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627857.186380:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627857/real 1778627857] req@ffff954342f49180 x1865013828567936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627857.186392:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828567936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627857.186400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627857.186403:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627857.186406:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828567936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627857.377751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627857.377814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627858.145751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627858.401769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627858.401822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627859.169797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627859.425769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627859.425819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627860.193789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627860.449775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627860.449842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627861.217781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627861.473783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627861.473856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627862.241756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627862.305730:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627862.305741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627862.305746:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627862.305754:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12196 00000100:00080000:0.0:1778627862.305764:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627862.305780:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627862.305825:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828568064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627862.305846:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627862.305849:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627862.305851:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627862.305854:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12196 00000100:00080000:0.0:1778627862.305857:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627862.305859:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627862.305867:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627862.305876:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828568192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627862.305883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627862.305886:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627862.305887:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627862.305889:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12196 00000100:00080000:0.0:1778627862.305891:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627862.305893:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627862.305896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627862.305899:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828568064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627862.305904:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828568320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627862.305914:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627862.305916:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627862.305917:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627862.305926:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12196 00000100:00080000:0.0:1778627862.305929:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627862.305931:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627862.305935:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627862.305946:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828568448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627862.305953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627862.305955:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627862.305956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627862.305959:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12196 00000100:00080000:0.0:1778627862.305962:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627862.305964:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627862.305966:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627862.305974:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627862.305981:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828568576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627862.306013:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627862/real 1778627862] req@ffff954342f4bb80 x1865013828568064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627862.306036:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828568064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627862.306044:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627862.306047:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627862.306050:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828568064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627862.306055:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627862.306073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627862.306080:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627862/real 1778627862] req@ffff954342f49880 x1865013828568192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627862.306088:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828568192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627862.306094:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627862.306096:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627862.306098:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627862.306116:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627862.306126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627862.306131:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627862/real 1778627862] req@ffff954342f4a300 x1865013828568320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627862.306138:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828568320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627862.306152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627862.306154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627862.306156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627862.306160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627862.306173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627862.306179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627862/real 1778627862] req@ffff954342f4b480 x1865013828568448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627862.306204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828568448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627862.306211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627862.306213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627862.306215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627862.306219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627862.306231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627862.306238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627862/real 1778627862] req@ffff954342f49c00 x1865013828568576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627862.306262:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828568576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627862.306267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627862.306269:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627862.306275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627862.497761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627862.497840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627863.265771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627863.521784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627863.521848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627864.289782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627864.545849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627864.545915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627865.313852:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627865.569788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627865.569957:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627866.337828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627866.593794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627866.593880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627867.361782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627867.425783:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627867.425791:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627867.425795:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627867.425803:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12201 00000100:00080000:0.0:1778627867.425813:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627867.425834:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627867.425863:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828568704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627867.425886:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627867.425890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627867.425891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627867.425896:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12201 00000100:00080000:0.0:1778627867.425900:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627867.425904:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627867.425913:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627867.425925:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828568832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627867.425932:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828568704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627867.425935:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627867.425938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627867.425940:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627867.425943:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12201 00000100:00080000:0.0:1778627867.425946:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627867.425949:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627867.425953:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627867.425961:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828568960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627867.425968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627867.425971:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627867.425979:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627867.425982:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12201 00000100:00080000:0.0:1778627867.425985:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627867.425989:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778627867.425989:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778627867.426005:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627867.426017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828569088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627867.426024:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627867/real 1778627867] req@ffff954342f48700 x1865013828568704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627867.426026:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627867.426029:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627867.426031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627867.426034:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12201 00000100:00080000:0.0:1778627867.426037:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627867.426041:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627867.426043:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828568704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778627867.426045:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627867.426053:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828569216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627867.426053:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627867.426058:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627867.426062:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828568704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627867.426084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627867.426106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627867.426119:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627867/real 1778627867] req@ffff954342f4b100 x1865013828568832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627867.426133:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828568832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627867.426141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627867.426145:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627867.426156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627867.426162:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627867.426184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627867.426196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627867/real 1778627867] req@ffff954342f49f80 x1865013828568960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627867.426204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828568960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627867.426209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627867.426211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627867.426213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828568960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627867.426217:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627867.426231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627867.426236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627867/real 1778627867] req@ffff954342f49500 x1865013828569088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627867.426243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828569088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627867.426248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627867.426250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627867.426252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627867.426255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627867.426264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627867.426276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627867/real 1778627867] req@ffff954342f4ad80 x1865013828569216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627867.426283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828569216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627867.426288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627867.426290:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627867.426292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627867.617770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627867.617844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627868.385775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627868.641801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627868.641901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627869.409747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627869.665793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627869.666354:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627870.433794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627870.689833:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627870.689955:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627871.457762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627871.713890:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627871.714136:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627872.481798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627872.545754:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627872.545763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627872.545768:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627872.545778:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12207 00000100:00080000:0.0:1778627872.545807:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627872.545815:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627872.545852:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828569344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627872.545882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627872.545887:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627872.545889:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627872.545893:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12207 00000100:00080000:0.0:1778627872.545897:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627872.545901:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627872.545911:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627872.545926:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828569472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627872.545931:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828569344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627872.545944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627872.545947:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627872.545949:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627872.545954:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12207 00000100:00080000:0.0:1778627872.545958:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627872.545961:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627872.545967:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627872.545980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828569600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627872.545990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778627872.545991:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627872.546000:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627872.546003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627872.546009:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12207 00000100:00080000:0.0:1778627872.546014:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627872.546019:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627872.546026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627872.546030:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627872/real 1778627872] req@ffff954342f48a80 x1865013828569344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627872.546043:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828569728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627872.546054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627872.546056:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828569344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627872.546063:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627872.546068:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627872.546072:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627872.546076:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12207 00000100:00080000:2.0:1778627872.546079:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778627872.546083:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627872.546084:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828569344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627872.546092:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627872.546098:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627872.546108:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627872.546113:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828569856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627872.546130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627872.546150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627872/real 1778627872] req@ffff954342f4a680 x1865013828569472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627872.546158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828569472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627872.546164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627872.546166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627872.546169:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627872.546173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627872.546184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627872.546190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627872/real 1778627872] req@ffff954342f4ad80 x1865013828569600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627872.546197:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828569600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627872.546202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627872.546204:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627872.546205:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627872.546209:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627872.546218:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627872.546223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627872/real 1778627872] req@ffff954342f49500 x1865013828569728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627872.546230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828569728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627872.546241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627872.546243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627872.546245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627872.546256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627872.546268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627872.546273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627872/real 1778627872] req@ffff954342f49f80 x1865013828569856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627872.546280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828569856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627872.546286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627872.546287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627872.546289:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828569856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627872.737957:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627872.738725:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627873.505821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627873.762112:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778627873.762118:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627874.529790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627874.785789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627874.785870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627875.553768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627875.809763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627875.809816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627876.577770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627876.833770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627876.833838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627877.601767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627877.665830:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627877.665841:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627877.665846:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627877.665857:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12212 00000100:00080000:2.0:1778627877.665868:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627877.665875:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627877.665924:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828569984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627877.665962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627877.665966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627877.665968:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627877.665974:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12212 00000100:00080000:2.0:1778627877.665979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627877.665985:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627877.665995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627877.666002:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828569984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627877.666012:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828570112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627877.666026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627877.666029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627877.666031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627877.666036:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12212 00000100:00080000:2.0:1778627877.666040:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627877.666044:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627877.666049:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627877.666060:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828570240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627877.666072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778627877.666073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627877.666077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627877.666079:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627877.666105:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12212 00000100:00080000:2.0:1778627877.666112:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627877.666117:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627877.666124:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627877.666134:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627877/real 1778627877] req@ffff954358789180 x1865013828569984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627877.666140:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828570368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627877.666167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627877.666171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778627877.666173:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627877.666177:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828569984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627877.666181:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12212 00000100:00080000:2.0:1778627877.666188:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627877.666196:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778627877.666200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1778627877.666202:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627877.666209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627877.666215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828569984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627877.666218:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828570496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627877.666259:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627877.666287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627877.666301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627877/real 1778627877] req@ffff954358788a80 x1865013828570112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627877.666316:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828570112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627877.666326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627877.666330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627877.666334:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627877.666340:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627877.666363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627877.666376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627877/real 1778627877] req@ffff95435878b800 x1865013828570240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627877.666389:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828570240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627877.666414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627877.666418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627877.666422:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627877.666428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627877.666454:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627877.666478:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627877/real 1778627877] req@ffff95435878a300 x1865013828570368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627877.666493:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828570368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627877.666503:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627877.666506:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627877.666510:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627877.666517:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627877.666539:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627877.666551:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627877/real 1778627877] req@ffff95435878ad80 x1865013828570496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627877.666564:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828570496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627877.666574:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627877.666577:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627877.666587:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627877.857803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627877.857863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627878.625791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627878.881773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627878.881840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627879.649744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627879.905775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627879.905813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627880.673783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627880.929806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627880.929869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627881.697760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627881.953734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627881.953833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627882.721739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627882.785794:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627882.785802:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627882.785805:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627882.785813:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12217 00000100:00080000:3.0:1778627882.785823:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627882.785843:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627882.785872:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828570624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627882.785900:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627882.785903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627882.785904:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627882.785908:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12217 00000100:00080000:3.0:1778627882.785911:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627882.785914:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627882.785922:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627882.785931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828570752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627882.785939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627882.785941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627882.785942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627882.785945:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12217 00000100:00080000:3.0:1778627882.785947:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627882.785949:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627882.785953:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627882.785954:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828570624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627882.785961:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828570880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627882.785969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627882.785971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627882.785978:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627882.785981:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12217 00000100:00080000:3.0:1778627882.785984:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627882.785987:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627882.785991:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627882.786002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828571008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627882.786008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627882.786011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627882.786012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627882.786014:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12217 00000100:00080000:3.0:1778627882.786016:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627882.786018:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627882.786021:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627882.786023:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627882.786031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828571136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627882.786058:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627882/real 1778627882] req@ffff95434701df80 x1865013828570624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627882.786080:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828570624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627882.786088:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627882.786091:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627882.786094:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828570624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627882.786100:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627882.786115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627882.786123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627882/real 1778627882] req@ffff95434701f480 x1865013828570752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627882.786131:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828570752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627882.786136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627882.786138:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627882.786146:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627882.786165:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627882.786179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627882.786186:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627882/real 1778627882] req@ffff95434701e680 x1865013828570880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627882.786193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828570880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627882.786198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627882.786200:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627882.786203:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828570880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627882.786206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627882.786216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627882.786221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627882/real 1778627882] req@ffff95434701d180 x1865013828571008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627882.786228:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828571008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627882.786233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627882.786234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627882.786236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627882.786239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627882.786248:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627882.786259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627882/real 1778627882] req@ffff95434701ca80 x1865013828571136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627882.786266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828571136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627882.786271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627882.786273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627882.786275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627882.977737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627882.977823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627883.745785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627884.001805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627884.001990:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627884.769804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627885.025792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627885.025891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627885.793822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627886.049786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627886.049860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627886.817753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627887.073761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627887.073828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627887.841774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627887.905773:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627887.905785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627887.905790:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627887.905801:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12222 00000100:00080000:1.0:1778627887.905813:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627887.905838:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627887.905872:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828571264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627887.905914:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627887.905918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627887.905920:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627887.905925:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12222 00000100:00080000:1.0:1778627887.905930:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627887.905935:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627887.905946:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627887.905962:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828571392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627887.905969:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828571264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627887.905979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627887.905983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627887.905985:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627887.905991:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12222 00000100:00080000:1.0:1778627887.905996:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627887.906001:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627887.906006:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627887.906017:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828571520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627887.906026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778627887.906030:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627887.906033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627887.906042:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627887.906049:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12222 00000100:00080000:1.0:1778627887.906055:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627887.906060:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627887.906065:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627887/real 1778627887] req@ffff954350196a00 x1865013828571264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627887.906068:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627887.906085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828571264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627887.906089:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828571648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627887.906101:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627887.906108:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627887.906111:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627887.906114:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828571264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627887.906119:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627887.906123:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627887.906129:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12222 00000100:00080000:1.0:1778627887.906134:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627887.906137:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571392:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627887.906141:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627887.906150:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778627887.906159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627887.906166:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828571776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627887.906179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627887/real 1778627887] req@ffff954350195880 x1865013828571392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627887.906190:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828571392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627887.906198:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627887.906201:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627887.906204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627887.906208:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627887.906221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627887.906226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627887/real 1778627887] req@ffff954350197b80 x1865013828571520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627887.906234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828571520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627887.906246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627887.906248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627887.906250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627887.906254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627887.906281:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627887.906288:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627887/real 1778627887] req@ffff954350195180 x1865013828571648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627887.906296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828571648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627887.906307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627887.906309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627887.906311:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627887.906323:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627887.906337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627887.906343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627887/real 1778627887] req@ffff954350195c00 x1865013828571776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627887.906350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828571776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627887.906356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627887.906358:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627887.906360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828571776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627888.097751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627888.097843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627888.865779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627889.121756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627889.121817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627889.889788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627890.145776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627890.145849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627890.913806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627891.169801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627891.169873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627891.937796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627892.193785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627892.193845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627892.961717:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778627893.025769:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627893.025777:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778627893.025781:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627893.025790:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12227 00000100:00080000:2.0:1778627893.025799:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627893.025804:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627893.025836:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828571904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627893.025858:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627893.025861:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778627893.025863:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627893.025865:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12227 00000100:00080000:2.0:1778627893.025868:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627893.025870:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627893.025876:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627893.025885:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828572032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627893.025892:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627893.025894:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778627893.025896:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627893.025898:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12227 00000100:00100000:0.0:1778627893.025905:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828571904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778627893.025913:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627893.025918:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627893.025924:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627893.025939:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828572160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627893.025950:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627893.025953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778627893.025955:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627893.025959:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12227 00000800:00000100:0.0:1778627893.025963:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778627893.025967:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627893.025973:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778627893.025979:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778627893.025993:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828572288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627893.025996:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627893/real 1778627893] req@ffff954358789f80 x1865013828571904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627893.026021:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778627893.026024:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627893.026026:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828571904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627893.026027:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778627893.026032:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12227 00000100:00080000:2.0:1778627893.026037:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778627893.026045:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778627893.026049:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1778627893.026051:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627893.026054:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627893.026059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828571904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627893.026068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627893.026068:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828572416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627893.026093:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627893.026105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627893/real 1778627893] req@ffff95435878a680 x1865013828572032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627893.026114:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828572032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627893.026121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627893.026124:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627893.026127:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627893.026146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627893.026156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627893.026174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627893/real 1778627893] req@ffff95435878ad80 x1865013828572160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627893.026183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828572160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627893.026189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627893.026191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627893.026193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627893.026203:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627893.026219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627893.026226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627893/real 1778627893] req@ffff95435878a300 x1865013828572288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627893.026233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828572288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627893.026238:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627893.026241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627893.026242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627893.026246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627893.026256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627893.026261:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627893/real 1778627893] req@ffff95435878b800 x1865013828572416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627893.026267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828572416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627893.026273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627893.026275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627893.026276:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627893.217754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627893.217810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627893.985743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627894.241790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627894.241845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627895.009788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627895.265736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627895.265798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627896.033730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627896.289759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627896.289848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627897.057731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627897.313768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627897.313844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627898.081778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627898.145786:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627898.145793:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627898.145797:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627898.145806:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12232 00000100:00080000:1.0:1778627898.145819:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627898.145826:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627898.145872:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828572544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627898.145894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627898.145897:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627898.145898:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627898.145902:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12232 00000100:00080000:1.0:1778627898.145905:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627898.145907:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627898.145915:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627898.145934:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828572672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627898.145945:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828572544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627898.145948:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627898.145954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627898.145956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627898.145961:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12232 00000100:00080000:1.0:1778627898.145966:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627898.145970:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627898.145976:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778627898.146001:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627898.146008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828572800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627898.146016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627898.146019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627898.146020:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627898.146025:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12232 00000100:00080000:1.0:1778627898.146029:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627898.146034:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627898.146035:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627898/real 1778627898] req@ffff954350195f80 x1865013828572544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627898.146043:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627898.146055:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828572544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627898.146059:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828572928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627898.146090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627898.146091:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627898.146096:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627898.146098:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627898.146100:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627898.146106:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828572544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627898.146108:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12232 00000100:00080000:1.0:1778627898.146114:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627898.146119:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627898.146123:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627898.146134:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627898.146135:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828573056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627898.146161:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627898.146173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627898/real 1778627898] req@ffff954350195c00 x1865013828572672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627898.146182:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828572672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627898.146189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627898.146191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627898.146193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627898.146197:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627898.146207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627898.146213:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627898/real 1778627898] req@ffff954350195180 x1865013828572800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627898.146228:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828572800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627898.146233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627898.146235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627898.146237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627898.146241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627898.146253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627898.146258:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627898/real 1778627898] req@ffff954350197b80 x1865013828572928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627898.146266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828572928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627898.146270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627898.146272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627898.146274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828572928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627898.146284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627898.146293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627898.146298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627898/real 1778627898] req@ffff954350195880 x1865013828573056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627898.146305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828573056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627898.146320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627898.146322:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627898.146324:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627898.337767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627898.337827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627899.105761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627899.361777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627899.361850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627900.129787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627900.385775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778627900.385847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627901.153759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627901.409770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627901.409854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627902.177754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627902.433795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627902.433871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627903.201723:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627903.265781:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627903.265806:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627903.265811:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627903.265820:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12237 00000100:00080000:1.0:1778627903.265832:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627903.265839:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627903.265870:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828573184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627903.265901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627903.265905:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627903.265907:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627903.265912:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12237 00000100:00080000:1.0:1778627903.265916:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627903.265920:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627903.265929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627903.265944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828573312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627903.265949:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828573184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627903.265984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627903.265988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627903.265990:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627903.265995:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12237 00000100:00080000:1.0:1778627903.266000:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627903.266005:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627903.266010:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627903.266025:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828573440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627903.266043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627903.266047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627903.266049:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627903.266054:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12237 00000100:00100000:0.0:1778627903.266058:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573312:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778627903.266062:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627903.266069:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627903.266076:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627903.266081:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627903.266095:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828573568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627903.266115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627903.266119:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627903.266121:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627903.266126:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12237 00000100:00100000:0.0:1778627903.266132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573568:192.168.204.102@tcp:8:kworker.0 00000100:00080000:1.0:1778627903.266132:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627903.266141:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627903.266148:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627903.266163:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828573696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627903.266250:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627903.457735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627903.457788:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627904.225743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627904.481771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627904.481842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627905.249788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627905.505742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627905.505832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627906.273747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627906.337951:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627906.337986:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627906.337999:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627906.338037:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627906.338067:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627906.338079:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627906.338089:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778627906.338090:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627903/real 1778627906] req@ffff954350195500 x1865013828573312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627906.338118:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828573312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627906.338130:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627906.338135:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627906.338147:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627906.338156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627903/real 1778627906] req@ffff954350194e00 x1865013828573440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627906.338171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828573440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627906.338180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627906.338184:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627906.338188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627906.338194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627903/real 1778627906] req@ffff954350196a00 x1865013828573184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627906.338209:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828573184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627906.338219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627906.338229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627906.338232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828573184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627906.338257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627903/real 1778627906] req@ffff954350194a80 x1865013828573696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627906.338270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828573696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627906.338280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627906.338284:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627906.338303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627906.338313:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627903/real 1778627906] req@ffff954350194380 x1865013828573568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627906.338325:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828573568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627906.338333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627906.338336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627906.338339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627906.529774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627906.529838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627907.297791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627907.553767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627907.553827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627908.321790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627908.385758:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627908.385768:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627908.385773:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627908.385784:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12242 00000100:00080000:1.0:1778627908.385807:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627908.385813:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627908.385842:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828573824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627908.385880:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627908.385884:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627908.385890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627908.385895:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12242 00000100:00080000:1.0:1778627908.385898:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627908.385902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627908.385912:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627908.385931:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828573952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627908.385936:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828573824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627908.385953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627908.385957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627908.385959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627908.385964:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12242 00000100:00080000:1.0:1778627908.385969:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627908.385973:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627908.385979:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627908.385992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828574080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627908.386003:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627908.386008:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627908.386020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627908.386023:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627908.386029:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12242 00000100:00080000:1.0:1778627908.386035:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627908.386040:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627908/real 1778627908] req@ffff954350197800 x1865013828573824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627908.386044:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627908.386053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627908.386070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828573824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627908.386071:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828574208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627908.386108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778627908.386108:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627908.386115:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627908.386117:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778627908.386119:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627908.386125:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828573824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627908.386128:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12242 00000100:00080000:1.0:1778627908.386136:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627908.386142:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627908.386147:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627908.386154:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778627908.386161:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828574336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778627908.386183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627908.386200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627908/real 1778627908] req@ffff954350194700 x1865013828573952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627908.386222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828573952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627908.386232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627908.386236:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627908.386239:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828573952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627908.386246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627908.386268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627908.386279:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627908/real 1778627908] req@ffff954350197480 x1865013828574080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627908.386292:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828574080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627908.386300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627908.386303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627908.386306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627908.386313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627908.386335:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627908.386345:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627908/real 1778627908] req@ffff954350196d80 x1865013828574208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627908.386357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828574208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627908.386381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627908.386385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627908.386388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627908.386407:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627908.386440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627908.386452:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627908/real 1778627908] req@ffff954350194000 x1865013828574336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627908.386462:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828574336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627908.386470:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627908.386474:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627908.386477:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627908.577756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627908.577824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627909.345795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627909.601771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627909.601821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627910.369743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627910.625747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627910.625811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627911.393789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627911.649771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627911.649849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627912.417760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627912.673746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627912.673893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627913.441863:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627913.505718:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627913.505726:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627913.505730:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627913.505740:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12248 00000100:00080000:1.0:1778627913.505751:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627913.505770:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627913.505807:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828574464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627913.505838:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627913.505841:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627913.505843:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627913.505846:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12248 00000100:00080000:1.0:1778627913.505850:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627913.505855:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627913.505865:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627913.505875:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828574464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627913.505883:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828574592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627913.505897:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627913.505901:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627913.505903:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627913.505907:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12248 00000100:00080000:1.0:1778627913.505912:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627913.505915:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627913.505920:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627913.505929:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828574720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627913.505936:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627913.505939:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627913.505941:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627913.505943:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627913.505946:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12248 00000100:00080000:1.0:1778627913.505956:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627913.505959:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627913.505965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627913.505974:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828574848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627913.505982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627913.505985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627913.505986:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778627913.505989:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627913/real 1778627913] req@ffff954350197100 x1865013828574464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627913.505992:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12248 00000100:00080000:1.0:1778627913.505998:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627913.506004:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627913.506007:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627913.506014:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828574464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627913.506016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828574976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627913.506036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627913.506042:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627913.506046:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828574464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627913.506071:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627913.506095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627913.506105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627913/real 1778627913] req@ffff954350194000 x1865013828574592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627913.506114:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828574592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627913.506120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627913.506122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627913.506124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627913.506129:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627913.506140:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627913.506145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627913/real 1778627913] req@ffff954350196d80 x1865013828574720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627913.506153:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828574720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627913.506158:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627913.506159:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627913.506161:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627913.506172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627913.506192:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627913.506203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627913/real 1778627913] req@ffff954350197480 x1865013828574848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627913.506215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828574848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627913.506224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627913.506227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627913.506231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627913.506236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627913.506253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627913.506260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627913/real 1778627913] req@ffff954350194700 x1865013828574976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627913.506267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828574976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627913.506273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627913.506275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627913.506277:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828574976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627913.697766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627913.697840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627914.465779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627914.721742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627914.721805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627915.489765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627915.745785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627915.745848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627916.513782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627916.769759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627916.769825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627917.537795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627917.793772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627917.793835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627918.561753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627918.625744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627918.625753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627918.625757:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627918.625765:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12253 00000100:00080000:1.0:1778627918.625774:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627918.625780:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627918.625817:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828575104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627918.625845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627918.625868:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627918.625870:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627918.625876:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12253 00000100:00080000:1.0:1778627918.625881:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627918.625886:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627918.625896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627918.625910:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828575104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627918.625913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828575232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627918.625929:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627918.625933:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627918.625935:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627918.625939:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12253 00000100:00080000:1.0:1778627918.625944:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627918.625948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627918.625954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627918.625967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828575360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627918.625978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778627918.625980:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627918.625984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627918.625987:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627918.625993:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12253 00000100:00080000:1.0:1778627918.625998:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627918.626003:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627918.626016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627918.626024:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627918/real 1778627918] req@ffff954350197800 x1865013828575104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627918.626031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828575488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627918.626052:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627918.626054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627918.626055:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627918.626059:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12253 00000100:00080000:1.0:1778627918.626063:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627918.626067:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828575104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627918.626068:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627918.626074:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627918.626085:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828575616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627918.626087:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627918.626098:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627918.626106:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828575104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627918.626135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627918.626162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627918.626181:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627918/real 1778627918] req@ffff954350194380 x1865013828575232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627918.626193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828575232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627918.626202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627918.626206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627918.626209:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627918.626216:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627918.626244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627918.626255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627918/real 1778627918] req@ffff954350194a80 x1865013828575360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627918.626277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828575360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627918.626286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627918.626289:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627918.626292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627918.626301:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627918.626329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627918.626341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627918/real 1778627918] req@ffff954350196a00 x1865013828575488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627918.626354:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828575488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627918.626362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627918.626365:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627918.626368:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627918.626374:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627918.626395:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627918.626406:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627918/real 1778627918] req@ffff954350194e00 x1865013828575616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627918.626418:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828575616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627918.626442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627918.626445:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627918.626447:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627918.817813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627918.817883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627919.585794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627919.841771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627919.841842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627920.609751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627920.865736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627920.865812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627921.633792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627921.889798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627921.889869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627922.657826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627922.913803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627922.913888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627923.681766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627923.745820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627923.745830:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627923.745836:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627923.745846:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12258 00000100:00080000:1.0:1778627923.745860:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627923.745866:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627923.745913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828575744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627923.745943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627923.745946:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627923.745947:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627923.745950:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12258 00000100:00080000:1.0:1778627923.745972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627923.745976:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627923.745987:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627923.745998:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828575744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778627923.745998:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828575872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627923.746013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627923.746015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627923.746017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627923.746020:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12258 00000100:00080000:1.0:1778627923.746022:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627923.746025:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627923.746028:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627923.746035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828576000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627923.746042:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627923.746045:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627923.746045:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627923.746048:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12258 00000100:00080000:1.0:1778627923.746065:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627923.746068:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627923.746071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627923.746078:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627923.746082:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828576128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627923.746100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627923.746103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627923.746104:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627923.746108:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12258 00000100:00080000:1.0:1778627923.746111:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627923.746114:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627923.746117:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627923.746127:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828576256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627923.746133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627923/real 1778627923] req@ffff954350195500 x1865013828575744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627923.746158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828575744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627923.746169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627923.746172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627923.746176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828575744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627923.746197:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627923.746216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627923.746225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627923/real 1778627923] req@ffff954350195880 x1865013828575872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627923.746234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828575872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627923.746247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627923.746249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627923.746252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828575872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627923.746256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627923.746269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627923.746276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627923/real 1778627923] req@ffff954350197b80 x1865013828576000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627923.746283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828576000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627923.746288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627923.746290:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627923.746292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627923.746296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627923.746308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627923.746313:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627923/real 1778627923] req@ffff954350195180 x1865013828576128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627923.746321:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828576128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627923.746326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627923.746327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627923.746334:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627923.746338:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627923.746351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627923.746356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627923/real 1778627923] req@ffff954350195c00 x1865013828576256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627923.746364:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828576256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627923.746370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627923.746372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627923.746373:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627923.937824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627923.937934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627924.705820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627924.961800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627924.961870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627925.729757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627925.985786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627925.985850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627926.753816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627927.009834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627927.009901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627927.777742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627928.033766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627928.033845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627928.801837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627928.865796:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627928.865807:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627928.865812:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627928.865821:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12263 00000100:00080000:3.0:1778627928.865834:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627928.865840:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627928.865883:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828576384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627928.865904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627928.865907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627928.865909:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627928.865912:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12263 00000100:00080000:3.0:1778627928.865915:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627928.865918:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627928.865926:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627928.865934:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828576512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627928.865940:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828576384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627928.865944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627928.865950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627928.865951:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627928.865954:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12263 00000100:00080000:3.0:1778627928.865956:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627928.865958:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627928.865962:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627928.865970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828576640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627928.865976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627928.865978:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627928.865979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627928.865987:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12263 00000100:00080000:3.0:1778627928.865990:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627928.865993:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627928.865998:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627928.866007:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627928.866008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828576768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627928.866017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627928.866019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627928.866020:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627928.866024:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12263 00000100:00080000:3.0:1778627928.866026:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627928.866029:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627928.866032:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627928.866046:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627928/real 1778627928] req@ffff95434701d880 x1865013828576384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627928.866046:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828576896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627928.866108:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828576384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627928.866120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627928.866125:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627928.866134:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828576384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627928.866153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627928.866171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627928.866180:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627928/real 1778627928] req@ffff95434701ea00 x1865013828576512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627928.866197:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828576512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627928.866203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627928.866205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627928.866208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627928.866212:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627928.866223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627928.866229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627928/real 1778627928] req@ffff95434701c000 x1865013828576640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627928.866237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828576640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627928.866242:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627928.866244:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627928.866245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627928.866249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627928.866260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627928.866288:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627928/real 1778627928] req@ffff95434701c700 x1865013828576768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627928.866297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828576768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627928.866303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627928.866305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627928.866307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627928.866310:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627928.866321:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627928.866327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627928/real 1778627928] req@ffff95434701f800 x1865013828576896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627928.866341:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828576896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627928.866362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627928.866364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627928.866366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828576896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627929.057748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627929.057808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627929.825749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627930.081736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627930.081799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627930.849788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627931.105773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627931.105848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627931.873773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627932.129793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627932.129822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627932.897748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627933.153795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627933.154017:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627933.921823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627933.985819:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627933.985831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627933.985837:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627933.985849:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12268 00000100:00080000:1.0:1778627933.985863:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627933.985871:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627933.985927:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828577024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627933.985988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627933.985994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627933.985997:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627933.986004:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12268 00000100:00080000:1.0:1778627933.986008:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627933.986014:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627933.986061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627933.986084:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828577152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627933.986103:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778627933.986109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828577024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627933.986110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627933.986116:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627933.986129:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12268 00000100:00080000:1.0:1778627933.986135:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627933.986140:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627933.986147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627933.986163:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828577280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627933.986176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627933.986195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627933.986196:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627933.986200:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12268 00000100:00080000:1.0:1778627933.986205:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627933.986208:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627933.986213:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778627933.986219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627933.986228:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828577408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627933.986240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627933.986260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627933.986274:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627933.986280:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12268 00000100:00080000:1.0:1778627933.986288:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627933.986294:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627933.986302:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627933.986307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627933/real 1778627933] req@ffff954350195f80 x1865013828577024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627933.986323:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828577536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627933.986346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828577024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627933.986395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627933.986420:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627933.986424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828577024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627933.986447:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627933.986469:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627933.986479:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627933/real 1778627933] req@ffff954350195c00 x1865013828577152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627933.986490:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828577152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627933.986498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627933.986500:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627933.986503:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627933.986508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627933.986520:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627933.986526:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627933/real 1778627933] req@ffff954350195180 x1865013828577280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627933.986536:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828577280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627933.986556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627933.986559:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627933.986561:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627933.986567:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627933.986583:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627933.986591:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627933/real 1778627933] req@ffff954350197b80 x1865013828577408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627933.986600:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828577408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627933.986607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627933.986609:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627933.986611:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627933.986616:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627933.986627:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627933.986633:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627933/real 1778627933] req@ffff954350195880 x1865013828577536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627933.986642:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828577536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627933.986649:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627933.986651:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627933.986658:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627934.177806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627934.177923:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627934.945743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627935.201806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627935.202006:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627935.969732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778627936.225823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627936.225903:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627936.993811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778627937.250311:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627937.250464:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627938.017795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627938.273889:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627938.274394:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627939.041810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627939.105811:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627939.105821:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627939.105827:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627939.105838:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12273 00000100:00080000:1.0:1778627939.105852:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627939.105862:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627939.105909:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828577664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627939.105946:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627939.105951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627939.105953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627939.105958:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12273 00000100:00080000:1.0:1778627939.105963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627939.105967:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627939.105978:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627939.105992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828577792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627939.106029:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627939.106047:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627939.106051:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627939.106057:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12273 00000100:00080000:1.0:1778627939.106065:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627939.106067:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828577664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627939.106076:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627939.106085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627939.106104:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828577920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627939.106119:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627939.106123:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627939.106126:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627939.106130:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12273 00000100:00080000:1.0:1778627939.106146:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627939.106150:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778627939.106159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778627939.106160:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627939.106182:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828578048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627939.106195:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627939.106208:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778627939.106214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627939/real 1778627939] req@ffff954350195500 x1865013828577664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627939.106214:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627939.106228:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12273 00000100:00080000:1.0:1778627939.106245:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778627939.106279:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828577664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627939.106281:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627939.106295:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627939.106313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627939.106324:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778627939.106327:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828578176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627939.106335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828577664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627939.106363:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627939.106399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627939.106415:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627939/real 1778627939] req@ffff954350194e00 x1865013828577792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627939.106431:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828577792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627939.106441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627939.106457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627939.106462:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627939.106494:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627939.106520:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627939.106533:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627939/real 1778627939] req@ffff954350196a00 x1865013828577920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627939.106547:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828577920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627939.106557:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627939.106561:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627939.106565:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828577920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627939.106571:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627939.106589:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627939.106599:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627939/real 1778627939] req@ffff954350194a80 x1865013828578048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627939.106612:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828578048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627939.106622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627939.106626:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627939.106629:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627939.106636:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627939.106719:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627939.106737:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627939/real 1778627939] req@ffff954350194380 x1865013828578176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627939.106752:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828578176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627939.106763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627939.106767:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627939.106771:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578176:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778627939.298066:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627939.298638:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778627940.065784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778627940.322366:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627940.322616:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627941.089786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627941.345842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627941.345914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627942.113782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627942.369796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627942.369859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627943.137793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627943.393757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627943.393940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627944.161784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627944.225799:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627944.225810:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627944.225816:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627944.225827:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12278 00000100:00080000:1.0:1778627944.225841:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627944.225850:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627944.225896:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828578304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627944.225933:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627944.225938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627944.225941:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627944.225946:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12278 00000100:00080000:1.0:1778627944.225951:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627944.225955:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627944.225967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627944.225982:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828578432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627944.226000:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828578304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627944.226001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627944.226027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627944.226030:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627944.226037:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12278 00000100:00080000:1.0:1778627944.226044:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627944.226049:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627944.226057:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627944.226074:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828578560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627944.226087:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627944.226092:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627944.226094:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778627944.226101:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627944.226108:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12278 00000100:00080000:1.0:1778627944.226115:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627944.226121:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627944.226128:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627944.226142:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828578688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778627944.226185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627944/real 1778627944] req@ffff954350197800 x1865013828578304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778627944.226186:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627944.226199:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627944.226205:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627944.226219:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12278 00000100:00080000:1.0:1778627944.226229:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627944.226236:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627944.226244:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828578304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778627944.226259:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778627944.226283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778627944.226285:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828578816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627944.226302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627944.226316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828578304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778627944.226373:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627944.226409:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627944.226426:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627944/real 1778627944] req@ffff954350194700 x1865013828578432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627944.226443:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828578432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627944.226454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627944.226459:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627944.226463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627944.226470:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778627944.226499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627944.226518:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627944/real 1778627944] req@ffff954350197480 x1865013828578560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627944.226559:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828578560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627944.226578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627944.226584:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627944.226591:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778627944.226603:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627944.226635:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627944.226649:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627944/real 1778627944] req@ffff954350196d80 x1865013828578688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627944.226727:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828578688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627944.226739:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627944.226743:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627944.226747:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778627944.226755:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778627944.226783:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778627944.226796:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627944/real 1778627944] req@ffff954350194000 x1865013828578816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627944.226810:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828578816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627944.226820:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778627944.226823:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778627944.226826:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828578816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627944.417758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627944.417869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627945.185769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627945.441812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627945.441929:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627946.209828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627946.465784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627946.465945:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627947.233798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627947.489728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627947.489798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627948.257752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627948.513759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627948.513855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627949.281786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627949.345794:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627949.345802:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627949.345805:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627949.345822:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12283 00000100:00080000:0.0:1778627949.345831:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627949.345836:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627949.345864:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828578944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627949.345888:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627949.345890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627949.345892:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627949.345895:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12283 00000100:00080000:0.0:1778627949.345897:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627949.345900:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627949.345907:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627949.345916:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828579072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627949.345927:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627949.345931:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627949.345933:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627949.345937:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12283 00000100:00100000:2.0:1778627949.345941:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828578944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627949.345943:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627949.345951:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627949.345956:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627949.345970:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828579200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627949.345993:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627949.345995:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627949.345997:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778627949.346002:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627949.346003:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12283 00000100:00080000:0.0:1778627949.346009:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627949.346012:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627949.346017:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627949.346027:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828579328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627949.346035:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778627949.346035:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627949/real 1778627949] req@ffff954342f4b100 x1865013828578944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778627949.346044:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627949.346047:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627949.346064:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12283 00000100:00080000:0.0:1778627949.346067:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627949.346071:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627949.346075:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627949.346077:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828578944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627949.346095:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828579456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627949.346099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627949.346107:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627949.346112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828578944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627949.346125:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627949.346144:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627949.346152:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627949/real 1778627949] req@ffff954342f48700 x1865013828579072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627949.346161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828579072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627949.346167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627949.346169:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627949.346171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627949.346190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627949.346210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627949.346216:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627949/real 1778627949] req@ffff954342f49c00 x1865013828579200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627949.346223:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828579200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627949.346228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627949.346229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627949.346231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627949.346235:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627949.346244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627949.346249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627949/real 1778627949] req@ffff954342f4b480 x1865013828579328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627949.346262:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828579328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627949.346268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627949.346269:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627949.346271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627949.346275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627949.346286:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627949.346291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627949/real 1778627949] req@ffff954342f4a300 x1865013828579456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627949.346298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828579456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627949.346303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627949.346305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627949.346307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627949.537742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627949.537825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627950.305794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627950.561797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627950.561972:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627951.329798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627951.585867:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627951.585949:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627952.353881:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627952.609819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627952.609959:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627953.377781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627953.633820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627953.633942:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627954.401780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627954.465868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627954.465881:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627954.465887:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627954.465899:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12288 00000100:00080000:1.0:1778627954.465918:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627954.465930:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627954.466017:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828579584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627954.466073:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627954.466079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627954.466081:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627954.466088:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12288 00000100:00080000:1.0:1778627954.466095:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627954.466102:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627954.466114:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627954.466137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828579712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627954.466145:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828579584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627954.466164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627954.466169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627954.466171:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627954.466176:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12288 00000100:00080000:1.0:1778627954.466202:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627954.466208:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627954.466216:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627954.466232:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828579840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627954.466246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627954.466251:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627954.466253:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627954.466258:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12288 00000100:00080000:1.0:1778627954.466262:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627954.466274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627954.466276:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627954.466284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627954.466301:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828579968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627954.466313:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627954.466318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627954.466320:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627954.466325:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12288 00000100:00080000:1.0:1778627954.466329:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627954.466334:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627954.466340:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627954.466341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627954/real 1778627954] req@ffff954350197100 x1865013828579584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778627954.466363:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828580096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627954.466428:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828579584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627954.466493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627954.466505:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627954.466514:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828579584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627954.466555:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627954.466586:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627954.466601:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627954/real 1778627954] req@ffff954350194000 x1865013828579712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627954.466618:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828579712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627954.466628:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627954.466632:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627954.466636:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627954.466644:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627954.466665:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627954.466707:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627954/real 1778627954] req@ffff954350196d80 x1865013828579840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627954.466722:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828579840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627954.466746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627954.466751:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627954.466755:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627954.466763:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627954.466790:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627954.466802:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627954/real 1778627954] req@ffff954350197480 x1865013828579968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627954.466815:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828579968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627954.466825:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627954.466829:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627954.466832:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828579968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627954.466839:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627954.466861:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627954.466873:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627954/real 1778627954] req@ffff954350194700 x1865013828580096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627954.466895:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828580096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627954.466910:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627954.466914:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627954.466924:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627954.657782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627954.657877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627955.425802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627955.681786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627955.681979:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627956.449825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627956.705845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627956.706026:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627957.473838:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627957.729774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627957.729974:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627958.497853:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627958.753803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627958.754003:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627959.521772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627959.585927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627959.585940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627959.585947:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627959.585961:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12294 00000100:00080000:3.0:1778627959.585981:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627959.585991:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627959.586056:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828580224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627959.586101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627959.586106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627959.586132:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627959.586139:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12294 00000100:00080000:3.0:1778627959.586145:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627959.586150:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627959.586166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627959.586185:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828580352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627959.586200:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627959.586204:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627959.586207:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627959.586213:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12294 00000100:00080000:3.0:1778627959.586217:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627959.586222:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627959.586228:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778627959.586241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828580224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778627959.586245:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828580480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627959.586280:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627959.586285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627959.586288:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627959.586296:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12294 00000100:00080000:3.0:1778627959.586301:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627959.586306:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627959.586312:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778627959.586341:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828580608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627959.586356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778627959.586368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778627959.586371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627959.586375:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627959.586384:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12294 00000100:00080000:3.0:1778627959.586394:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627959.586402:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627959.586409:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778627959.586442:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828580736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627959.586445:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627959/real 1778627959] req@ffff95434701d500 x1865013828580224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627959.586542:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828580224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627959.586560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627959.586567:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627959.586573:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828580224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627959.586584:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627959.586611:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627959.586624:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627959/real 1778627959] req@ffff95434701c380 x1865013828580352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627959.586639:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828580352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627959.586650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627959.586654:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627959.586726:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627959.586762:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627959.586792:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627959.586807:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627959/real 1778627959] req@ffff95434701f100 x1865013828580480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627959.586823:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828580480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627959.586833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627959.586837:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627959.586841:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627959.586878:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627959.586917:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627959.586937:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627959/real 1778627959] req@ffff95434701f800 x1865013828580608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627959.586960:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828580608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627959.586971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627959.586975:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627959.586979:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627959.586987:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627959.587012:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627959.587038:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627959/real 1778627959] req@ffff95434701c700 x1865013828580736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627959.587057:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828580736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627959.587069:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627959.587074:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627959.587078:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627959.777797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627959.777992:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627960.545737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627960.801866:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627960.801979:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627961.569773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627961.825854:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627961.825969:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627962.593854:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627962.849862:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627962.849963:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627963.617894:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627963.873894:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627963.873990:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627964.641873:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627964.705894:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627964.705912:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627964.705920:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627964.705935:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12299 00000100:00080000:0.0:1778627964.705953:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627964.705966:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627964.706017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828580864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627964.706059:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627964.706064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627964.706067:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627964.706072:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12299 00000100:00080000:0.0:1778627964.706077:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627964.706082:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627964.706093:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627964.706114:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828580992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627964.706178:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627964.706185:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627964.706190:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627964.706206:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12299 00000100:00100000:2.0:1778627964.706207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828580864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627964.706220:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627964.706229:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627964.706239:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627964.706262:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828581120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627964.706277:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627964.706283:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627964.706287:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627964.706292:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12299 00000100:00080000:0.0:1778627964.706298:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627964.706302:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627964.706307:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627964.706322:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828581248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778627964.706343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627964.706344:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627964.706355:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627964.706359:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627964.706371:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12299 00000100:00080000:0.0:1778627964.706401:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627964.706413:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627964.706418:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627964/real 1778627964] req@ffff954342f49880 x1865013828580864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778627964.706435:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627964.706459:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828580864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778627964.706465:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828581376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778627964.706521:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627964.706534:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627964.706542:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828580864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627964.706587:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627964.706624:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627964.706766:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627964/real 1778627964] req@ffff954342f4bb80 x1865013828580992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627964.706790:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828580992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627964.706802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627964.706807:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627964.706812:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828580992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627964.706821:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627964.706854:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627964.706869:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627964/real 1778627964] req@ffff954342f49180 x1865013828581120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627964.706884:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828581120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627964.706893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627964.706897:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627964.706900:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627964.706909:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627964.706930:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627964.706941:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627964/real 1778627964] req@ffff954342f48000 x1865013828581248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627964.706955:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828581248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627964.706975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627964.706980:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627964.706985:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627964.706992:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627964.707019:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627964.707031:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627964/real 1778627964] req@ffff954342f48380 x1865013828581376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627964.707046:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828581376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627964.707058:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627964.707062:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627964.707065:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627964.897882:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627964.897907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627965.665792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627965.921822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627965.921927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627966.689790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627966.945843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627966.945940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627967.713804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627967.969803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627967.969906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627968.737806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627968.993817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627968.993897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627969.761789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627969.825773:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627969.825783:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627969.825788:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627969.825799:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12304 00000100:00080000:0.0:1778627969.825813:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627969.825822:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627969.825869:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828581504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627969.825906:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627969.825911:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627969.825913:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627969.825919:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12304 00000100:00080000:0.0:1778627969.825923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627969.825927:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627969.825938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627969.825971:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828581632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627969.825987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627969.825991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627969.825994:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627969.826000:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12304 00000100:00080000:0.0:1778627969.826006:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627969.826016:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778627969.826016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828581504:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778627969.826029:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627969.826046:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828581760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627969.826060:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627969.826064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627969.826067:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627969.826071:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12304 00000100:00080000:0.0:1778627969.826076:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627969.826080:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627969.826086:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627969.826099:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828581888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627969.826112:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627969.826116:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627969.826117:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627969.826130:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12304 00000100:00080000:0.0:1778627969.826136:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627969.826141:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627969.826149:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627969.826167:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828582016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627969.826179:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627969.826259:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627969.826307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627969.826370:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627970.017809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627970.017942:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627970.785737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627971.041726:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627971.041806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627971.809799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627972.065802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627972.065934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627972.833780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778627972.898049:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778627972.898100:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778627972.898122:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627972.898185:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627972.898195:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627972.898204:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778627972.898212:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778627972.898297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627969/real 1778627972] req@ffff954342f48380 x1865013828581760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627972.898347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828581760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627972.898370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627972.898380:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627972.898388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627972.898403:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627969/real 1778627972] req@ffff954342f48000 x1865013828581888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627972.898429:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828581888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627972.898445:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627972.898454:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627972.898460:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627972.898474:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627969/real 1778627972] req@ffff954342f49180 x1865013828582016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627972.898494:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828582016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627972.898528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627972.898551:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627972.898558:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627972.898572:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627969/real 1778627972] req@ffff954342f4aa00 x1865013828581504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627972.898637:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828581504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627972.898653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627972.898659:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627972.898666:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828581504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627972.898735:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627969/real 1778627972] req@ffff954342f48e00 x1865013828581632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627972.898765:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828581632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627972.898783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627972.898791:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627972.898799:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828581632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627973.089815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627973.089934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627973.857791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627974.113808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627974.113999:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627974.881764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778627974.945883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627974.945900:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778627974.945906:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627974.945921:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12309 00000100:00080000:3.0:1778627974.945934:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627974.945943:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778627974.945987:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828582144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627974.946025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627974.946029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778627974.946039:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627974.946045:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12309 00000100:00080000:3.0:1778627974.946049:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627974.946054:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627974.946065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627974.946082:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828582272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627974.946096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627974.946101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778627974.946116:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778627974.946123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828582144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778627974.946126:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12309 00000100:00080000:3.0:1778627974.946136:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627974.946146:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627974.946153:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778627974.946172:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828582400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627974.946190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627974.946196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778627974.946200:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627974.946208:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12309 00000100:00080000:3.0:1778627974.946216:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627974.946225:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627974.946234:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778627974.946246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778627974.946261:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828582528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778627974.946284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778627974.946290:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778627974.946294:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778627974.946302:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12309 00000100:00080000:3.0:1778627974.946309:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778627974.946317:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778627974.946328:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778627974.946335:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627974/real 1778627974] req@ffff95434701c000 x1865013828582144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778627974.946378:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828582656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627974.946388:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828582144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627974.946436:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627974.946446:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627974.946456:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828582144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627974.946472:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627974.946521:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627974.946573:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627974/real 1778627974] req@ffff95434701ea00 x1865013828582272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627974.946601:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828582272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627974.946618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627974.946625:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627974.946632:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627974.946741:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627974.946788:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627974.946812:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627974/real 1778627974] req@ffff95434701d880 x1865013828582400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627974.946839:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828582400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627974.946893:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627974.946900:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627974.946908:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627974.946921:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627974.946979:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627974.947004:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627974/real 1778627974] req@ffff95434701ca80 x1865013828582528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627974.947028:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828582528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627974.947043:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627974.947049:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627974.947054:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627974.947066:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627974.947105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627974.947127:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627974/real 1778627974] req@ffff95434701d180 x1865013828582656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627974.947151:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828582656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778627999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627974.947169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627974.947176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627974.947196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627975.137777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627975.137935:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627975.905785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627976.161809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627976.161933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627976.929802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627977.185815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627977.185909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627977.953779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627978.209765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627978.209946:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627978.977762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627979.233792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627979.233950:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627980.001788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627980.065834:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627980.065844:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627980.065850:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627980.065861:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12314 00000100:00080000:1.0:1778627980.065877:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627980.065888:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627980.065936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828582784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627980.065975:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627980.065980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627980.065983:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627980.065988:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12314 00000100:00080000:1.0:1778627980.065992:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627980.065997:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627980.066007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627980.066021:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828582912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627980.066034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627980.066038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627980.066040:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627980.066044:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12314 00000100:00080000:1.0:1778627980.066047:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627980.066050:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627980.066077:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627980.066092:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828583040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627980.066105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627980.066109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627980.066112:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627980.066117:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12314 00000100:00080000:1.0:1778627980.066122:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778627980.066137:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828582784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627980.066144:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627980.066157:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627980.066177:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828583168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627980.066192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627980.066197:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627980.066199:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627980.066203:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12314 00000100:00080000:1.0:1778627980.066210:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627980.066214:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627980.066220:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627980.066234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627980.066266:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828583296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627980.066288:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627980/real 1778627980] req@ffff954350197800 x1865013828582784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627980.066330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828582784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627980.066346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627980.066352:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627980.066358:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828582784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627980.066370:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627980.066397:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627980.066412:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627980/real 1778627980] req@ffff954350194380 x1865013828582912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627980.066427:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828582912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627980.066438:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627980.066473:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627980.066494:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828582912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627980.066508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627980.066544:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627980.066560:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627980/real 1778627980] req@ffff954350194a80 x1865013828583040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627980.066576:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828583040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627980.066586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627980.066590:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627980.066594:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627980.066640:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627980.066749:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627980.066775:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627980/real 1778627980] req@ffff954350196a00 x1865013828583168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627980.066802:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828583168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627980.066819:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627980.066840:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627980.066848:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627980.066858:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627980.066889:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627980.066917:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627980/real 1778627980] req@ffff954350194e00 x1865013828583296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627980.066933:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828583296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627980.066944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627980.066948:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627980.066952:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627980.257781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627980.257910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627981.025789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627981.281816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627981.281994:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627982.049784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627982.305802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627982.305890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627983.073806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627983.329824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627983.329927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627984.097778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627984.353797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627984.353934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627985.121771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627985.185792:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627985.185803:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627985.185827:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627985.185838:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12319 00000100:00080000:1.0:1778627985.185853:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627985.185863:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627985.185905:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828583424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627985.185944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627985.185949:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627985.185952:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627985.185957:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12319 00000100:00080000:1.0:1778627985.185961:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627985.185965:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627985.185975:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627985.185989:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828583552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627985.186002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627985.186006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627985.186008:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627985.186011:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12319 00000100:00080000:1.0:1778627985.186015:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627985.186019:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627985.186024:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627985.186036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828583680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627985.186068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828583424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627985.186069:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627985.186086:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627985.186090:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627985.186097:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12319 00000100:00080000:1.0:1778627985.186102:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627985.186107:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627985.186114:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627985.186133:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828583808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627985.186144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627985.186149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627985.186151:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627985.186156:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12319 00000100:00080000:1.0:1778627985.186160:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627985.186163:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627985.186170:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778627985.186173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778627985.186191:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828583936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627985.186228:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627985/real 1778627985] req@ffff954350195500 x1865013828583424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627985.186255:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828583424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627985.186270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627985.186275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627985.186281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828583424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627985.186290:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627985.186315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627985.186379:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627985/real 1778627985] req@ffff954350195880 x1865013828583552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627985.186406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828583552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627985.186424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627985.186431:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627985.186437:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627985.186477:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627985.186519:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627985.186561:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627985/real 1778627985] req@ffff954350197b80 x1865013828583680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627985.186588:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828583680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627985.186604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627985.186611:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627985.186617:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627985.186626:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627985.186657:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627985.186713:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627985/real 1778627985] req@ffff954350195180 x1865013828583808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627985.186732:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828583808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627985.186752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627985.186757:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627985.186761:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627985.186769:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627985.186797:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627985.186811:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627985/real 1778627985] req@ffff954350195c00 x1865013828583936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627985.186825:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828583936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627985.186835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627985.186839:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627985.186843:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828583936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627985.377775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627985.377964:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627986.145816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627986.401828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627986.401947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627987.169807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627987.425819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627987.425953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627988.193784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627988.449766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627988.449925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627989.217859:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627989.473802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627989.473928:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627990.241815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778627990.305781:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627990.305791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778627990.305796:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627990.305809:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12324 00000100:00080000:1.0:1778627990.305847:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627990.305857:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778627990.305902:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828584064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627990.305940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627990.305945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778627990.305948:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627990.305954:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12324 00000100:00080000:1.0:1778627990.305958:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627990.305962:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627990.305974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627990.305989:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828584192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627990.306009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627990.306016:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778627990.306020:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627990.306029:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12324 00000100:00080000:1.0:1778627990.306036:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627990.306045:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627990.306056:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778627990.306080:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828584320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627990.306093:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828584064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778627990.306123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627990.306140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778627990.306144:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627990.306153:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12324 00000100:00080000:1.0:1778627990.306160:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778627990.306166:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627990.306176:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778627990.306197:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828584448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778627990.306210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778627990.306214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778627990.306217:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778627990.306221:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12324 00000100:00080000:1.0:1778627990.306225:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778627990.306229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778627990.306236:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778627990.306271:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778627990.306293:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828584576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627990.306339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627990/real 1778627990] req@ffff954350195f80 x1865013828584064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627990.306373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828584064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627990.306410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627990.306417:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627990.306423:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828584064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627990.306434:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627990.306472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627990.306487:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627990/real 1778627990] req@ffff954350195c00 x1865013828584192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627990.306503:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828584192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627990.306514:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627990.306519:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627990.306523:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627990.306559:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627990.306579:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627990.306591:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627990/real 1778627990] req@ffff954350195180 x1865013828584320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627990.306605:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828584320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627990.306615:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627990.306618:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627990.306622:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627990.306641:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627990.306668:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627990.306734:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627990/real 1778627990] req@ffff954350197b80 x1865013828584448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627990.306751:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828584448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627990.306762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627990.306767:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627990.306771:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627990.306779:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627990.306803:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627990.306814:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627990/real 1778627990] req@ffff954350195880 x1865013828584576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627990.306828:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828584576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627990.306839:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627990.306842:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627990.306846:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627990.497845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627990.497894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627991.265825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627991.521835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627991.521940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627992.289938:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627992.545812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627992.545978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627993.313882:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627993.569808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627993.569908:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627994.337808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627994.593865:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627994.593952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627995.361919:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778627995.425847:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627995.425859:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778627995.425864:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627995.425876:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12329 00000100:00080000:0.0:1778627995.425903:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627995.425912:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778627995.425970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828584704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627995.426009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627995.426012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778627995.426015:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627995.426019:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12329 00000100:00080000:0.0:1778627995.426022:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627995.426026:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627995.426037:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627995.426052:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828584832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627995.426062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627995.426066:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778627995.426068:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627995.426071:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12329 00000100:00080000:0.0:1778627995.426074:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627995.426077:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627995.426108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778627995.426131:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828584960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627995.426158:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828584704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778627995.426172:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627995.426176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778627995.426178:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627995.426184:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12329 00000100:00080000:0.0:1778627995.426190:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627995.426196:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627995.426204:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778627995.426220:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828585088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778627995.426230:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778627995.426233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778627995.426235:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778627995.426244:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12329 00000800:00000100:2.0:1778627995.426244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778627995.426254:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778627995.426259:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778627995.426264:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778627995.426275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828585216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778627995.426291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627995/real 1778627995] req@ffff954342f4bb80 x1865013828584704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627995.426310:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828584704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627995.426321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627995.426325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627995.426328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828584704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778627995.426335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627995.426351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627995.426359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627995/real 1778627995] req@ffff954342f49880 x1865013828584832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627995.426367:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828584832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627995.426373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627995.426384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627995.426386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627995.426409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778627995.426422:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627995.426428:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627995/real 1778627995] req@ffff954342f4a300 x1865013828584960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627995.426453:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828584960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627995.426458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627995.426461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627995.426463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828584960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778627995.426468:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627995.426482:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627995.426489:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627995/real 1778627995] req@ffff954342f4b480 x1865013828585088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627995.426496:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828585088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627995.426502:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627995.426504:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627995.426506:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627995.426510:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778627995.426520:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778627995.426526:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778627995/real 1778627995] req@ffff954342f49c00 x1865013828585216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778627995.426541:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828585216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778627995.426556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778627995.426559:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778627995.426561:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778627995.617752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627995.617974:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627996.385785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627996.641856:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627996.641992:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627997.409818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627997.665775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627997.665899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627998.433782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627998.689775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627998.689863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778627999.457861:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778627999.713848:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778627999.713965:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628000.481794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628000.545844:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628000.545859:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628000.545865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628000.545881:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12335 00000100:00080000:1.0:1778628000.545898:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628000.545909:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628000.545967:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828585344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628000.546065:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628000.546071:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628000.546075:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628000.546081:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12335 00000100:00080000:1.0:1778628000.546088:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628000.546093:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628000.546106:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628000.546126:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828585472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628000.546141:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628000.546146:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628000.546148:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628000.546154:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12335 00000100:00080000:1.0:1778628000.546159:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628000.546163:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628000.546171:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628000.546172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828585344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628000.546185:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828585600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628000.546199:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628000.546203:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628000.546205:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628000.546211:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12335 00000100:00080000:1.0:1778628000.546227:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628000.546233:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628000.546241:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628000.546260:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828585728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628000.546267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628000.546283:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628000.546287:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628000.546290:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628000.546298:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12335 00000100:00080000:1.0:1778628000.546302:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628000.546308:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628000.546314:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628000.546327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628000/real 1778628000] req@ffff954350195500 x1865013828585344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628000.546333:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828585856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628000.546406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828585344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628000.546423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628000.546430:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628000.546437:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828585344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628000.546447:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628000.546472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628000.546485:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628000/real 1778628000] req@ffff954350194e00 x1865013828585472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628000.546500:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828585472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628000.546523:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628000.546527:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628000.546532:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628000.546563:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628000.546587:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628000.546599:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628000/real 1778628000] req@ffff954350196a00 x1865013828585600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628000.546614:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828585600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628000.546623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628000.546627:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628000.546631:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628000.546638:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628000.546656:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628000.546666:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628000/real 1778628000] req@ffff954350194a80 x1865013828585728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628000.546723:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828585728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628000.546733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628000.546737:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628000.546751:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628000.546759:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628000.546787:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628000.546800:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628000/real 1778628000] req@ffff954350194380 x1865013828585856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628000.546815:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828585856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628000.546835:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628000.546839:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628000.546843:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828585856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628000.737829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628000.737892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628001.505761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628001.761899:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628001.762151:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628002.529739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628002.785981:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628002.786135:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628003.553748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628003.810048:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628003.810163:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628004.577812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628004.833888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628004.834020:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628005.601816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628005.665866:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628005.665877:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628005.665883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628005.665894:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12340 00000100:00080000:0.0:1778628005.665908:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628005.665917:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628005.666013:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828585984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628005.666073:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628005.666080:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628005.666085:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628005.666093:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12340 00000100:00080000:0.0:1778628005.666101:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628005.666108:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628005.666123:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628005.666165:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828586112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628005.666206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828585984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628005.666240:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628005.666249:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628005.666253:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628005.666260:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12340 00000100:00080000:0.0:1778628005.666268:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628005.666274:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628005.666284:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778628005.666295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628005.666309:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828586240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628005.666344:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628005/real 1778628005] req@ffff954342f48700 x1865013828585984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628005.666345:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628005.666365:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628005.666374:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628005.666387:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12340 00000100:00100000:2.0:1778628005.666390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828585984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628005.666400:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628005.666414:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778628005.666416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778628005.666426:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:2.0:1778628005.666426:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628005.666437:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828585984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628005.666453:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828586368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628005.666481:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628005.666485:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628005.666488:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628005.666493:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12340 00000100:00100000:2.0:1778628005.666503:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586112:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778628005.666518:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628005.666528:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628005.666537:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778628005.666554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628005.666561:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828586496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628005.666577:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628005/real 1778628005] req@ffff954342f4b100 x1865013828586112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628005.666597:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828586112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628005.666618:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628005.666624:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628005.666628:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628005.666636:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628005.666655:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628005.666666:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628005/real 1778628005] req@ffff954342f49f80 x1865013828586240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628005.666723:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828586240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628005.666734:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628005.666738:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628005.666741:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628005.666749:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628005.666772:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628005.666785:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628005/real 1778628005] req@ffff954342f49500 x1865013828586368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628005.666811:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828586368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628005.666822:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628005.666825:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628005.666829:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628005.666852:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628005.666875:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628005.666886:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628005/real 1778628005] req@ffff954342f4ad80 x1865013828586496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628005.666901:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828586496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628005.666911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628005.666915:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628005.666918:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628005.857976:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628005.858148:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628006.625828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628006.882293:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628006.882350:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628007.649826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628007.905794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628007.905912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628008.673788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628008.929825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628008.929915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628009.697810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628009.953830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628009.953956:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628010.721816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628010.785865:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628010.785905:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628010.785910:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628010.785922:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12345 00000100:00080000:3.0:1778628010.785936:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628010.785945:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628010.785999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828586624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628010.786038:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628010.786043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628010.786045:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628010.786050:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12345 00000100:00080000:3.0:1778628010.786055:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628010.786059:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628010.786070:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628010.786084:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828586752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628010.786098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628010.786103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628010.786106:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628010.786111:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12345 00000100:00080000:3.0:1778628010.786115:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628010.786119:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628010.786125:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628010.786132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828586624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628010.786142:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828586880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628010.786167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628010.786171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628010.786174:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628010.786180:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12345 00000100:00080000:3.0:1778628010.786186:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628010.786192:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628010.786198:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628010.786217:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828587008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628010.786240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628010.786251:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628010.786256:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628010.786259:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628010.786266:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12345 00000100:00080000:3.0:1778628010.786272:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628010.786277:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628010.786284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628010.786300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628010/real 1778628010] req@ffff95434701e680 x1865013828586624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628010.786301:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828587136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628010.786389:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828586624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628010.786404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628010.786410:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628010.786415:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828586624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628010.786425:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628010.786454:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628010.786468:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628010/real 1778628010] req@ffff95434701f480 x1865013828586752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628010.786484:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828586752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628010.786494:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628010.786498:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628010.786502:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628010.786529:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628010.786548:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628010.786558:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628010/real 1778628010] req@ffff95434701df80 x1865013828586880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628010.786572:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828586880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628010.786598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628010.786602:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628010.786606:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828586880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628010.786613:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628010.786648:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628010.786659:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628010/real 1778628010] req@ffff95434701d180 x1865013828587008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628010.786727:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828587008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628010.786738:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628010.786742:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628010.786745:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628010.786752:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628010.786788:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628010.786809:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628010/real 1778628010] req@ffff95434701ca80 x1865013828587136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628010.786866:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828587136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628010.786883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628010.786900:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628010.786919:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628010.977772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628010.977962:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628011.745951:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628012.001871:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628012.001983:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628012.769780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628013.025821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628013.025898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628013.793849:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628014.049806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628014.049964:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628014.817744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628015.073807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628015.073912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628015.841820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628015.905775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628015.905787:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628015.905793:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628015.905805:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12350 00000100:00080000:3.0:1778628015.905833:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628015.905842:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628015.905898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828587264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628015.905932:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628015.905937:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628015.905940:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628015.905946:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12350 00000100:00080000:3.0:1778628015.905950:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628015.905954:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628015.905965:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628015.905982:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828587392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628015.905998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628015.906002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628015.906006:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628015.906011:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828587264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628015.906015:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12350 00000100:00080000:3.0:1778628015.906027:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628015.906033:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628015.906039:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628015.906057:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828587520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628015.906081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628015.906096:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628015.906099:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628015.906104:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12350 00000100:00080000:3.0:1778628015.906111:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628015.906122:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778628015.906123:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778628015.906137:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628015.906156:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828587648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628015.906170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628015.906175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628015.906178:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628015.906184:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12350 00000100:00100000:2.0:1778628015.906192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628015/real 1778628015] req@ffff95434701d880 x1865013828587264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628015.906194:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628015.906208:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628015.906222:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628015.906243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828587264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628015.906245:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828587776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628015.906266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628015.906283:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628015.906294:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828587264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628015.906328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628015.906362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628015.906378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628015/real 1778628015] req@ffff95434701ea00 x1865013828587392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628015.906394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828587392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628015.906405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628015.906409:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628015.906426:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628015.906434:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628015.906460:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628015.906472:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628015/real 1778628015] req@ffff95434701c000 x1865013828587520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628015.906487:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828587520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628015.906497:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628015.906501:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628015.906505:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628015.906512:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628015.906530:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628015.906540:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628015/real 1778628015] req@ffff95434701c700 x1865013828587648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628015.906554:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828587648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628015.906563:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628015.906566:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628015.906570:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628015.906576:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628015.906593:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628015.906614:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628015/real 1778628015] req@ffff95434701f800 x1865013828587776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628015.906628:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828587776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628015.906639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628015.906642:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628015.906646:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828587776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628016.097831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628016.097986:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628016.865822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628017.121810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628017.121945:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628017.889834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628018.145905:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628018.145906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628018.913789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628019.169811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628019.169971:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628019.937830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628020.193816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628020.193918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628020.961830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628021.025800:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628021.025816:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628021.025823:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628021.025838:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12355 00000100:00080000:3.0:1778628021.025857:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628021.025867:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628021.025934:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828587904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628021.026012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628021.026020:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628021.026025:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628021.026034:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12355 00000100:00080000:3.0:1778628021.026043:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628021.026052:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628021.026071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628021.026090:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828587904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628021.026104:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828588032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628021.026130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628021.026138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628021.026143:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628021.026151:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12355 00000100:00080000:3.0:1778628021.026183:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628021.026194:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628021.026203:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778628021.026216:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628021.026226:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828588160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628021.026243:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628021.026248:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628021.026252:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628021.026257:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12355 00000100:00080000:3.0:1778628021.026275:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628021.026282:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628021.026290:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628021.026292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628021/real 1778628021] req@ffff95434701f100 x1865013828587904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628021.026315:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828588288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628021.026330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628021.026335:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828587904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628021.026337:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628021.026344:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628021.026358:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12355 00000100:00080000:3.0:1778628021.026367:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628021.026369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778628021.026374:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778628021.026383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:3.0:1778628021.026385:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628021.026394:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828587904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628021.026403:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828588416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628021.026438:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628021.026476:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628021.026527:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628021/real 1778628021] req@ffff95434701c380 x1865013828588032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628021.026545:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828588032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628021.026557:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628021.026561:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628021.026566:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628021.026574:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628021.026600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628021.026611:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628021/real 1778628021] req@ffff95434701d500 x1865013828588160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628021.026627:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828588160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628021.026664:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628021.026701:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628021.026706:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628021.026716:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628021.026747:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628021.026761:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628021/real 1778628021] req@ffff95434701f800 x1865013828588288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628021.026776:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828588288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628021.026796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628021.026801:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628021.026805:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628021.026813:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628021.026838:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628021.026849:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628021/real 1778628021] req@ffff95434701c700 x1865013828588416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628021.026863:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828588416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628021.026874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628021.026877:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628021.026881:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628021.217787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628021.217933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628021.985913:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628022.241794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628022.241952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628023.009814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628023.265808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628023.265910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628024.033859:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628024.289834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628024.289977:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628025.057829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628025.313818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628025.313873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628026.081809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628026.145851:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628026.145865:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628026.145873:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628026.145887:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12360 00000100:00080000:1.0:1778628026.145919:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628026.145930:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628026.146004:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828588544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628026.146066:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628026.146074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628026.146078:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628026.146088:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12360 00000100:00080000:1.0:1778628026.146096:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628026.146103:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628026.146120:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628026.146142:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828588672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628026.146171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828588544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628026.146202:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628026.146207:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628026.146209:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628026.146217:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12360 00000100:00080000:1.0:1778628026.146224:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628026.146229:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628026.146237:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628026.146255:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828588800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628026.146269:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628026.146274:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628026.146276:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628026.146281:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12360 00000100:00080000:1.0:1778628026.146285:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628026.146289:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628026.146294:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778628026.146308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628026.146314:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828588928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628026.146328:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628026.146333:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628026.146336:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628026.146341:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12360 00000100:00080000:1.0:1778628026.146354:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628026.146361:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628026.146370:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628026.146378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628026/real 1778628026] req@ffff954350197800 x1865013828588544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628026.146390:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828589056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628026.146446:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828588544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628026.146506:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628026.146515:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628026.146521:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828588544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628026.146583:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628026.146621:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628026.146640:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628026/real 1778628026] req@ffff954350194700 x1865013828588672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628026.146658:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828588672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628026.146710:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628026.146716:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628026.146721:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628026.146729:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628026.146763:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628026.146785:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628026/real 1778628026] req@ffff954350197480 x1865013828588800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628026.146809:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828588800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628026.146828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628026.146835:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628026.146843:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628026.146869:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628026.146902:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628026.146917:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628026/real 1778628026] req@ffff954350196d80 x1865013828588928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628026.146933:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828588928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628026.146944:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628026.146948:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628026.146952:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828588928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628026.146960:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628026.146979:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628026.146989:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628026/real 1778628026] req@ffff954350194000 x1865013828589056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628026.147004:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828589056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628026.147014:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628026.147018:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628026.147021:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628026.337844:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628026.337936:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628027.105788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628027.361766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628027.361875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628028.129818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628028.385799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628028.385898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628029.153844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628029.409815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628029.409943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628030.177796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628030.433828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628030.433882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628031.201787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628031.265820:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628031.265831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628031.265834:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628031.265842:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12365 00000100:00080000:3.0:1778628031.265852:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628031.265858:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628031.265889:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828589184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628031.265913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628031.265927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628031.265929:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628031.265932:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12365 00000100:00080000:3.0:1778628031.265935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628031.265938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628031.265946:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628031.265955:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828589312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628031.265964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628031.265966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628031.265968:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628031.265972:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12365 00000100:00100000:2.0:1778628031.265973:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828589184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628031.265974:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628031.265980:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628031.265984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628031.265992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828589440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628031.265998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628031.266001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628031.266002:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628031.266005:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12365 00000100:00080000:3.0:1778628031.266006:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628031.266009:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628031.266017:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628031.266024:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828589568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628031.266031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628031.266033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628031.266034:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628031.266036:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12365 00000100:00080000:3.0:1778628031.266040:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778628031.266043:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628031.266047:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628031.266051:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628031.266059:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828589696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628031.266080:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628031/real 1778628031] req@ffff95434701c000 x1865013828589184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628031.266097:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828589184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628031.266105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628031.266108:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628031.266111:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828589184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628031.266128:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628031.266142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628031.266150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628031/real 1778628031] req@ffff95434701ea00 x1865013828589312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628031.266157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828589312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628031.266164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628031.266166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628031.266168:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628031.266171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628031.266180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628031.266185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628031/real 1778628031] req@ffff95434701d880 x1865013828589440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628031.266201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828589440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628031.266207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628031.266209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628031.266211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628031.266215:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628031.266227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628031.266233:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628031/real 1778628031] req@ffff95434701ca80 x1865013828589568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628031.266240:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828589568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628031.266245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628031.266247:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628031.266249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628031.266252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628031.266261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628031.266266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628031/real 1778628031] req@ffff95434701d180 x1865013828589696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628031.266273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828589696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628031.266287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628031.266289:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628031.266291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628031.457732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628031.457912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628032.225802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628032.481779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628032.481888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628033.249779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628033.505794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628033.505867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628034.273771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628034.529779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628034.529890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628035.297786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628035.553813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628035.553908:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628036.321808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628036.385789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628036.385799:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628036.385805:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628036.385816:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12370 00000100:00080000:3.0:1778628036.385830:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628036.385839:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628036.385884:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828589824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628036.385923:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628036.385927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628036.385930:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628036.385935:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12370 00000100:00080000:3.0:1778628036.385957:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628036.385963:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628036.385976:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628036.385994:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828589952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628036.386011:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628036.386014:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828589824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628036.386017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628036.386025:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628036.386033:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12370 00000100:00080000:3.0:1778628036.386038:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628036.386042:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628036.386048:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628036.386063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828590080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628036.386077:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628036.386081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628036.386083:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628036.386088:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12370 00000100:00080000:3.0:1778628036.386094:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628036.386099:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628036.386105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628036.386120:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828590208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628036.386157:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589952:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778628036.386171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628036.386180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628036.386192:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628036.386201:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12370 00000100:00080000:3.0:1778628036.386211:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628036.386219:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628036.386230:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628036.386241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778628036.386260:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828590336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628036.386282:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628036.386320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628036.577792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628036.577915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628037.345779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628037.601875:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628037.602009:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628038.369780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628038.625795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628038.625928:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628039.393780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778628039.457987:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778628039.458038:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778628039.458059:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628039.458104:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628039.458113:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628039.458122:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628039.458144:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778628039.458179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628036/real 1778628039] req@ffff95434701ca80 x1865013828590336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628039.458216:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828590336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628039.458231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628039.458238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628039.458260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628039.458299:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628036/real 1778628039] req@ffff95434701d180 x1865013828590208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628039.458319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828590208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628039.458330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628039.458334:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628039.458339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628039.458346:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628036/real 1778628039] req@ffff95434701f480 x1865013828589952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628039.458360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828589952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628039.458370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628039.458374:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628039.458378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828589952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628039.458385:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628036/real 1778628039] req@ffff95434701e680 x1865013828590080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628039.458398:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828590080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628039.458406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628039.458410:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628039.458422:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628039.458430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628036/real 1778628039] req@ffff95434701df80 x1865013828589824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628039.458445:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828589824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628039.458454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628039.458458:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628039.458462:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828589824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628039.649809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628039.649917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628040.417776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628040.673800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628040.673893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628041.441787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628041.505789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628041.505801:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628041.505806:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628041.505818:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12376 00000100:00080000:3.0:1778628041.505832:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628041.505841:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628041.505885:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828590464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628041.505921:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628041.505926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628041.505929:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628041.505934:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12376 00000100:00080000:3.0:1778628041.505938:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628041.505942:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628041.505974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628041.505988:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828590464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628041.505997:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828590592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628041.506014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628041.506018:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628041.506021:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628041.506026:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12376 00000100:00080000:3.0:1778628041.506032:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628041.506037:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628041.506043:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628041.506056:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828590720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628041.506068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628041.506073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628041.506076:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628041.506080:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12376 00000100:00080000:3.0:1778628041.506084:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628041.506088:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628041.506093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628041.506109:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828590848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628041.506117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628041.506139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628041.506146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628041.506160:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628041.506171:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12376 00000100:00080000:3.0:1778628041.506182:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778628041.506193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628041/real 1778628041] req@ffff95434701d880 x1865013828590464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628041.506193:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628041.506210:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628041.506281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828590464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628041.506289:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828590976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628041.506344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628041.506360:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628041.506370:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828590464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628041.506427:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628041.506476:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628041.506503:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628041/real 1778628041] req@ffff95434701ea00 x1865013828590592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628041.506549:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828590592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628041.506566:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628041.506573:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628041.506579:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628041.506591:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628041.506634:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628041.506656:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628041/real 1778628041] req@ffff95434701c000 x1865013828590720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628041.506739:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828590720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628041.506757:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628041.506764:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628041.506770:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628041.506782:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628041.506821:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628041.506842:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628041/real 1778628041] req@ffff95434701c700 x1865013828590848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628041.506864:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828590848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628041.506901:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628041.506908:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628041.506915:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628041.506926:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628041.506967:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628041.506988:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628041/real 1778628041] req@ffff95434701f800 x1865013828590976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628041.507011:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828590976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628041.507027:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628041.507034:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628041.507039:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828590976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628041.697811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628041.697908:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628042.465771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628042.721803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628042.721915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628043.489779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628043.745825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628043.745911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628044.513792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628044.769846:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628044.770004:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628045.537784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628045.793801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628045.793887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628046.561774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628046.625773:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628046.625783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628046.625788:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628046.625800:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12381 00000100:00080000:3.0:1778628046.625813:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628046.625822:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628046.625867:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828591104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628046.625904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628046.625909:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628046.625912:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628046.625917:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12381 00000100:00080000:3.0:1778628046.625922:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628046.625926:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628046.625937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628046.625951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828591232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628046.625987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628046.626005:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628046.626009:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628046.626015:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12381 00000100:00080000:3.0:1778628046.626021:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628046.626027:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628046.626036:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628046.626043:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828591104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628046.626056:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828591360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628046.626070:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628046.626075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628046.626077:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628046.626082:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12381 00000100:00080000:3.0:1778628046.626087:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628046.626091:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628046.626098:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628046.626113:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828591488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628046.626146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628046.626151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628046.626153:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628046.626158:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12381 00000100:00080000:3.0:1778628046.626172:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628046.626177:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628046.626184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628046.626198:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828591616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628046.626212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628046.626271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628046/real 1778628046] req@ffff95434701d500 x1865013828591104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628046.626297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828591104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628046.626324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628046.626330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628046.626336:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828591104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628046.626347:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628046.626395:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628046.626419:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628046/real 1778628046] req@ffff95434701c380 x1865013828591232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628046.626446:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828591232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628046.626464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628046.626472:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628046.626479:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628046.626518:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628046.626551:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628046.626566:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628046/real 1778628046] req@ffff95434701f100 x1865013828591360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628046.626581:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828591360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628046.626592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628046.626596:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628046.626600:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628046.626618:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628046.626643:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628046.626654:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628046/real 1778628046] req@ffff95434701f800 x1865013828591488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628046.626713:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828591488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628046.626725:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628046.626730:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628046.626733:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628046.626741:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628046.626765:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628046.626777:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628046/real 1778628046] req@ffff95434701c700 x1865013828591616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628046.626792:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828591616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628046.626802:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628046.626806:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628046.626809:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628046.817764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628046.817889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628047.585805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628047.841795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628047.841953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628048.609773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628048.865801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628048.865882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628049.633728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628049.889728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628049.889788:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628050.657732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628050.913748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628050.913791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628051.681735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628051.745756:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628051.745763:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628051.745766:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628051.745773:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12386 00000100:00080000:0.0:1778628051.745782:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628051.745788:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628051.745819:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828591744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628051.745841:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628051.745853:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628051.745855:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628051.745859:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12386 00000100:00080000:0.0:1778628051.745862:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628051.745866:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628051.745875:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628051.745883:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828591744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628051.745886:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828591872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628051.745897:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628051.745899:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628051.745900:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628051.745903:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12386 00000100:00080000:0.0:1778628051.745905:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628051.745907:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628051.745910:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628051.745919:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828592000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628051.745925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628051.745928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628051.745930:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628051.745932:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12386 00000100:00080000:0.0:1778628051.745942:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628051.745944:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778628051.745950:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778628051.745954:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628051.745965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828592128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628051.745973:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628051.745975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628051.745977:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628051.745979:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12386 00000100:00080000:0.0:1778628051.745983:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778628051.745983:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628051/real 1778628051] req@ffff954342f4a680 x1865013828591744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628051.745990:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628051.745997:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628051.746005:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828592256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628051.746009:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828591744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628051.746035:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628051.746039:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628051.746043:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828591744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628051.746061:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628051.746077:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628051.746085:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628051/real 1778628051] req@ffff954342f48a80 x1865013828591872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628051.746093:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828591872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628051.746099:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628051.746101:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628051.746104:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828591872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628051.746107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628051.746117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628051.746122:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628051/real 1778628051] req@ffff954342f4ad80 x1865013828592000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628051.746137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828592000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628051.746142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628051.746145:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628051.746146:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628051.746150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628051.746163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628051.746168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628051/real 1778628051] req@ffff954342f49500 x1865013828592128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628051.746176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828592128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628051.746181:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628051.746183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628051.746185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628051.746188:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628051.746197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628051.746201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628051/real 1778628051] req@ffff954342f49f80 x1865013828592256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628051.746208:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828592256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628051.746221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628051.746223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628051.746226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628051.937731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628051.937797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628052.705723:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628052.961735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628052.961815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628053.729729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628053.985768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628053.985831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628054.753833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628055.009778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628055.009942:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628055.777807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628056.033829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628056.034018:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628056.801856:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628056.865803:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628056.865815:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628056.865821:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628056.865837:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12391 00000100:00080000:0.0:1778628056.865854:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628056.865867:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628056.865951:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828592384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628056.866030:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628056.866035:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628056.866038:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628056.866045:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12391 00000100:00080000:0.0:1778628056.866061:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628056.866068:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628056.866080:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628056.866099:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828592512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628056.866112:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628056.866117:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628056.866120:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628056.866127:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12391 00000100:00080000:0.0:1778628056.866132:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628056.866136:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628056.866142:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628056.866155:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828592640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628056.866167:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828592384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628056.866174:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628056.866180:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628056.866182:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628056.866188:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12391 00000100:00080000:0.0:1778628056.866203:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628056.866209:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628056.866218:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628056.866235:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828592768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628056.866247:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628056.866252:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628056.866254:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628056.866260:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12391 00000800:00000100:2.0:1778628056.866269:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628056.866272:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628056.866280:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628056.866287:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628056.866301:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828592896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628056.866327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628056/real 1778628056] req@ffff954342f4b100 x1865013828592384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628056.866353:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828592384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628056.866367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628056.866373:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628056.866379:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828592384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628056.866388:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628056.866412:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628056.866425:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628056/real 1778628056] req@ffff954342f48700 x1865013828592512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628056.866440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828592512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628056.866463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628056.866468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628056.866473:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628056.866505:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628056.866531:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628056.866543:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628056/real 1778628056] req@ffff954342f49c00 x1865013828592640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628056.866557:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828592640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628056.866567:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628056.866571:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628056.866575:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628056.866581:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628056.866600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628056.866609:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628056/real 1778628056] req@ffff954342f4b480 x1865013828592768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628056.866623:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828592768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628056.866632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628056.866636:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628056.866648:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628056.866656:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628056.866732:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628056.866747:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628056/real 1778628056] req@ffff954342f4a300 x1865013828592896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628056.866763:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828592896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628056.866773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628056.866777:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628056.866781:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828592896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628057.057838:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628057.058017:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628057.825809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628058.081834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628058.081997:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628058.849795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628059.105785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628059.105959:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628059.873768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628060.129822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628060.129959:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628060.897788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628061.153790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628061.153920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628061.921793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628061.985759:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628061.985766:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628061.985770:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628061.985778:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12396 00000100:00080000:1.0:1778628061.985787:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628061.985793:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628061.985834:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828593024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628061.985855:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628061.985857:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628061.985859:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628061.985862:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12396 00000100:00080000:1.0:1778628061.985874:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628061.985888:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628061.985896:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628061.985910:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828593152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628061.985911:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828593024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628061.985925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628061.985928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628061.985929:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628061.985932:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12396 00000100:00080000:1.0:1778628061.985935:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628061.985938:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628061.985941:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628061.985949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828593280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628061.985955:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628061.985958:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628061.985959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628061.985962:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12396 00000100:00080000:1.0:1778628061.985965:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628061.985967:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628061.985970:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778628061.985984:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628061.985988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828593408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628061.985994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628061.986000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628061.986002:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628061.986005:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12396 00000100:00080000:1.0:1778628061.986010:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628061.986012:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628061.986016:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628061.986026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828593536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628061.986026:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628061/real 1778628061] req@ffff954350197100 x1865013828593024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628061.986069:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828593024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628061.986078:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628061.986082:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628061.986092:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828593024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628061.986110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628061.986138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628061.986146:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628061/real 1778628061] req@ffff954350194000 x1865013828593152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628061.986155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828593152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628061.986161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628061.986163:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628061.986165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628061.986168:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628061.986178:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628061.986183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628061/real 1778628061] req@ffff954350196d80 x1865013828593280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628061.986190:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828593280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628061.986195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628061.986197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628061.986199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628061.986202:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628061.986212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628061.986224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628061/real 1778628061] req@ffff954350197480 x1865013828593408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628061.986231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828593408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628061.986237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628061.986239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628061.986241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628061.986244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628061.986255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628061.986259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628061/real 1778628061] req@ffff954350194700 x1865013828593536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628061.986267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828593536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628061.986272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628061.986274:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628061.986276:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628062.177775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628062.177841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628062.945867:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628063.201882:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628063.202062:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628063.969763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628064.225812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628064.225884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628064.993850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628065.249819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628065.249870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628066.017771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628066.273850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628066.273947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628067.041802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628067.105885:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628067.105896:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628067.105901:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628067.105912:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12401 00000100:00080000:3.0:1778628067.105928:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628067.105939:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628067.105992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828593664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628067.106039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628067.106044:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628067.106046:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628067.106052:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12401 00000100:00080000:3.0:1778628067.106058:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628067.106063:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628067.106074:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628067.106112:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828593664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628067.106112:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828593792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628067.106136:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628067.106150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628067.106153:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628067.106158:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12401 00000100:00080000:3.0:1778628067.106163:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628067.106168:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628067.106174:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628067.106190:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828593920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628067.106201:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628067.106205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628067.106207:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628067.106212:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12401 00000100:00080000:3.0:1778628067.106216:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628067.106220:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628067.106226:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778628067.106230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628067.106246:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828594048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628067.106265:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628067.106271:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628067.106275:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628067.106296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628067/real 1778628067] req@ffff95434701c000 x1865013828593664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628067.106297:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12401 00000100:00080000:3.0:1778628067.106313:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628067.106325:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628067.106335:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828593664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778628067.106337:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778628067.106376:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628067.106384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:3.0:1778628067.106390:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828594176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628067.106397:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828593664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628067.106448:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628067.106486:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628067.106522:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628067/real 1778628067] req@ffff95434701ea00 x1865013828593792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628067.106546:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828593792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628067.106562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628067.106568:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628067.106574:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628067.106621:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628067.106664:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628067.106784:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628067/real 1778628067] req@ffff95434701d880 x1865013828593920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628067.106843:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828593920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628067.106860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628067.106866:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628067.106873:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828593920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628067.106886:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628067.106949:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628067.106972:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628067/real 1778628067] req@ffff95434701df80 x1865013828594048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628067.106988:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828594048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628067.106999:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628067.107003:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628067.107007:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628067.107015:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628067.107040:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628067.107051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628067/real 1778628067] req@ffff95434701e680 x1865013828594176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628067.107083:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828594176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628067.107109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628067.107113:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628067.107117:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628067.297854:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628067.297932:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628068.065770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628068.322148:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628068.322868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628069.089801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628069.345847:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628069.346595:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628070.113776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628070.370121:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628070.370336:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628071.137757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628071.393752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628071.393823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628072.161744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628072.225845:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628072.225853:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628072.225857:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628072.225864:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12406 00000100:00080000:2.0:1778628072.225874:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628072.225880:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628072.225912:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828594304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628072.225935:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628072.225938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628072.225939:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628072.225943:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12406 00000100:00080000:2.0:1778628072.225945:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628072.225947:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628072.225955:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628072.225963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828594432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628072.225975:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828594304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628072.225994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628072.225998:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628072.226000:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628072.226005:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12406 00000100:00080000:2.0:1778628072.226011:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628072.226015:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628072.226021:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778628072.226033:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628072.226037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828594560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628072.226048:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628072.226050:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628072.226052:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628072.226056:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12406 00000100:00080000:2.0:1778628072.226059:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628072.226063:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628072.226066:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628072/real 1778628072] req@ffff954358788a80 x1865013828594304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778628072.226068:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628072.226081:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828594688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628072.226087:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828594304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628072.226093:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628072.226100:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628072.226103:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628072.226109:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12406 00000100:00080000:0.0:1778628072.226110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628072.226116:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628072.226119:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628072.226121:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828594304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628072.226125:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628072.226130:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628072.226139:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828594816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628072.226150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628072.226176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628072.226190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628072/real 1778628072] req@ffff954358789180 x1865013828594432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628072.226203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828594432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628072.226212:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628072.226216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628072.226219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628072.226237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628072.226259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628072.226269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628072/real 1778628072] req@ffff954358788e00 x1865013828594560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628072.226277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828594560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628072.226283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628072.226285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628072.226288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628072.226292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628072.226304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628072.226309:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628072/real 1778628072] req@ffff954358789c00 x1865013828594688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628072.226316:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828594688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628072.226322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628072.226324:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628072.226325:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628072.226339:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628072.226349:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628072.226354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628072/real 1778628072] req@ffff954358789880 x1865013828594816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628072.226368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828594816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628072.226374:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628072.226376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628072.226378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828594816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628072.417736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628072.417811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628073.185756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628073.441829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628073.441989:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628074.209736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628074.465742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628074.465794:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628075.233803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628075.489762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628075.489885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628076.257805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628076.513793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628076.513916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628077.281756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628077.345856:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628077.345880:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628077.345884:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628077.345893:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12411 00000100:00080000:1.0:1778628077.345904:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628077.345910:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628077.345968:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828594944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628077.346018:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628077.346020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628077.346022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628077.346026:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12411 00000100:00080000:1.0:1778628077.346029:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628077.346033:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628077.346041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628077.346053:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828595072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628077.346057:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828594944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628077.346070:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628077.346078:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628077.346079:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628077.346082:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12411 00000100:00080000:1.0:1778628077.346085:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628077.346088:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628077.346091:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628077.346099:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828595200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628077.346106:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628077.346108:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628077.346109:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628077.346112:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12411 00000800:00000100:0.0:1778628077.346116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628077.346122:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628077.346126:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628077.346131:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628077.346139:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828595328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628077.346147:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628077.346149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628077/real 1778628077] req@ffff954350197800 x1865013828594944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628077.346150:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628077.346153:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628077.346160:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12411 00000100:00080000:1.0:1778628077.346165:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628077.346169:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628077.346173:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628077.346176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828594944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628077.346182:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828595456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628077.346192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628077.346198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628077.346202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828594944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628077.346220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628077.346234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628077.346242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628077/real 1778628077] req@ffff954350194380 x1865013828595072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628077.346257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828595072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628077.346263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628077.346266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628077.346268:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628077.346272:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628077.346284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628077.346290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628077/real 1778628077] req@ffff954350194a80 x1865013828595200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628077.346297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828595200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628077.346302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628077.346303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628077.346305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628077.346309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628077.346318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628077.346324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628077/real 1778628077] req@ffff954350196a00 x1865013828595328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628077.346331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828595328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628077.346345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628077.346347:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628077.346349:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628077.346353:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628077.346365:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628077.346371:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628077/real 1778628077] req@ffff954350194e00 x1865013828595456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628077.346378:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828595456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628077.346384:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628077.346385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628077.346387:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628077.537775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628077.537864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628078.305726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628078.561783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628078.561920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628079.329766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628079.585814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628079.585965:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628080.353795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628080.609852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628080.609935:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628081.377770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628081.633800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628081.634086:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628082.401806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628082.465779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628082.465788:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628082.465792:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628082.465801:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12416 00000100:00080000:3.0:1778628082.465812:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628082.465819:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628082.465867:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828595584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628082.465893:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628082.465896:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628082.465898:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628082.465902:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12416 00000100:00080000:3.0:1778628082.465905:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628082.465908:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628082.465916:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628082.465928:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828595712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628082.465938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628082.465954:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828595584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628082.465954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628082.465956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628082.465966:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12416 00000100:00080000:3.0:1778628082.465971:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628082.465974:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628082.465978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628082.465990:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828595840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628082.465999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628082.466002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628082.466004:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628082.466007:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12416 00000100:00080000:3.0:1778628082.466010:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628082.466014:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628082.466017:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628082.466027:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828595968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628082.466034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628082.466038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628082.466042:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778628082.466044:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628082.466047:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12416 00000100:00080000:3.0:1778628082.466052:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628082.466054:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628082.466067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628082.466079:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828596096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628082.466093:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628082/real 1778628082] req@ffff95434701f480 x1865013828595584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628082.466118:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828595584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628082.466142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628082.466148:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628082.466153:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828595584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628082.466163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628082.466193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628082.466206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628082/real 1778628082] req@ffff95434701d180 x1865013828595712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628082.466217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828595712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628082.466225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628082.466228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628082.466230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628082.466253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628082.466267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628082.466274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628082/real 1778628082] req@ffff95434701ca80 x1865013828595840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628082.466283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828595840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628082.466290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628082.466292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628082.466295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628082.466307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628082.466323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628082.466331:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628082/real 1778628082] req@ffff95434701e680 x1865013828595968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628082.466341:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828595968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628082.466347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628082.466350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628082.466353:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828595968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628082.466357:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628082.466368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628082.466375:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628082/real 1778628082] req@ffff95434701df80 x1865013828596096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628082.466384:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828596096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628082.466390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628082.466393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628082.466395:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628082.657827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628082.657952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628083.425808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628083.681803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628083.681842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628084.449776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628084.705789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628084.705944:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628085.473787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628085.729796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628085.729882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628086.497800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628086.753803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628086.753914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628087.521809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628087.585840:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628087.585852:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628087.585873:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628087.585885:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12422 00000100:00080000:1.0:1778628087.585899:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628087.585908:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628087.585950:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828596224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628087.585989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628087.585994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628087.585997:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628087.586002:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12422 00000100:00080000:1.0:1778628087.586006:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628087.586010:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628087.586023:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628087.586038:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828596352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628087.586054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628087.586059:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828596224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628087.586060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628087.586067:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628087.586080:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12422 00000100:00080000:1.0:1778628087.586084:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628087.586088:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628087.586095:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628087.586117:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828596480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628087.586159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628087.586168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628087.586177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628087.586182:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628087.586192:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12422 00000100:00080000:1.0:1778628087.586219:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628087.586229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628087/real 1778628087] req@ffff954350195500 x1865013828596224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628087.586232:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628087.586246:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628087.586270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828596224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628087.586279:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828596608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628087.586311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628087.586337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628087.586348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628087.586351:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628087.586353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778628087.586364:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12422 00000100:00100000:0.0:1778628087.586369:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828596224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628087.586373:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628087.586387:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628087.586395:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628087.586413:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828596736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628087.586417:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628087.586482:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628087.586506:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628087/real 1778628087] req@ffff954350195880 x1865013828596352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628087.586523:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828596352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628087.586534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628087.586539:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628087.586543:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628087.586551:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628087.586571:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628087.586582:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628087/real 1778628087] req@ffff954350197b80 x1865013828596480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628087.586611:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828596480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628087.586622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628087.586626:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628087.586632:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628087.586644:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628087.586735:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628087.586758:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628087/real 1778628087] req@ffff954350195180 x1865013828596608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628087.586779:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828596608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628087.586790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628087.586794:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628087.586798:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628087.586822:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628087.586872:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628087.586894:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628087/real 1778628087] req@ffff954350195c00 x1865013828596736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628087.586919:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828596736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628087.586961:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628087.586968:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628087.586973:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628087.777815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628087.777925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628088.545817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628088.801783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628088.801903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628089.569796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628089.825801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628089.825898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628090.593786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628090.849790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628090.849943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628091.617749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628091.873728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628091.873786:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628092.641751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628092.705731:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628092.705740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628092.705744:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628092.705753:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12427 00000100:00080000:1.0:1778628092.705764:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628092.705772:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628092.705807:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828596864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628092.705832:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628092.705835:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628092.705836:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628092.705839:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12427 00000100:00080000:1.0:1778628092.705842:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628092.705845:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628092.705852:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628092.705863:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828596992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628092.705872:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628092.705876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778628092.705877:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828596864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628092.705878:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628092.705886:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12427 00000100:00080000:1.0:1778628092.705888:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628092.705890:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628092.705893:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628092.705917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828597120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628092.705923:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628092.705926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628092.705927:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628092.705930:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12427 00000100:00080000:1.0:1778628092.705933:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628092.705937:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628092.705939:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628092.705945:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628092.705954:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828597248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628092.705960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628092.705963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628092.705964:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628092.705967:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12427 00000100:00080000:1.0:1778628092.705970:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628092.705972:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628092/real 1778628092] req@ffff954350195f80 x1865013828596864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628092.705975:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628092.705982:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628092.705996:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828597376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628092.706000:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828596864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628092.706022:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628092.706026:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628092.706029:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828596864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628092.706049:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628092.706063:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628092.706071:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628092/real 1778628092] req@ffff954350195c00 x1865013828596992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628092.706079:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828596992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628092.706091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628092.706094:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628092.706096:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828596992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628092.706100:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628092.706113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628092.706120:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628092/real 1778628092] req@ffff954350195180 x1865013828597120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628092.706127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828597120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628092.706132:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628092.706134:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628092.706135:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628092.706139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628092.706149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628092.706154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628092/real 1778628092] req@ffff954350197b80 x1865013828597248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628092.706162:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828597248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628092.706170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628092.706173:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628092.706183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628092.706190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628092.706223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628092.706236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628092/real 1778628092] req@ffff954350195880 x1865013828597376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628092.706247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828597376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628092.706264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628092.706266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628092.706268:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628092.897790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628092.897832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628093.665797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628093.921793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628093.921917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628094.689732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628094.945741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628094.945771:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628095.713730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628095.969729:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628095.969824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628096.737723:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628096.993755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628096.993799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628097.761789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628097.825758:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628097.825769:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628097.825774:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628097.825786:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12432 00000100:00080000:1.0:1778628097.825799:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628097.825808:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628097.825850:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828597504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628097.825882:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628097.825887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628097.825889:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628097.825894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12432 00000100:00080000:1.0:1778628097.825898:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628097.825901:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628097.825911:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628097.825923:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828597632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628097.825957:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628097.825961:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628097.825963:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628097.825967:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12432 00000100:00080000:1.0:1778628097.825971:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628097.825975:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628097.826003:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628097.826025:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828597760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628097.826037:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828597504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628097.826060:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628097.826067:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628097.826071:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628097.826081:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12432 00000100:00080000:1.0:1778628097.826108:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628097.826141:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628097.826152:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628097.826171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628097.826184:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828597888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628097.826200:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628097.826205:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628097.826208:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628097.826214:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12432 00000100:00080000:1.0:1778628097.826221:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628097.826230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628097/real 1778628097] req@ffff954350195500 x1865013828597504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628097.826230:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628097.826244:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628097.826269:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828598016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628097.826279:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828597504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628097.826309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628097.826316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628097.826336:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828597504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628097.826351:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628097.826385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628097.826424:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628097/real 1778628097] req@ffff954350194e00 x1865013828597632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628097.826438:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828597632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628097.826447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628097.826451:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628097.826455:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628097.826493:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628097.826515:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628097.826525:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628097/real 1778628097] req@ffff954350196a00 x1865013828597760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628097.826537:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828597760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628097.826545:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628097.826548:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628097.826552:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628097.826558:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628097.826575:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628097.826595:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628097/real 1778628097] req@ffff954350194a80 x1865013828597888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628097.826607:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828597888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628097.826616:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628097.826619:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628097.826622:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828597888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628097.826628:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628097.826645:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628097.826653:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628097/real 1778628097] req@ffff954350194380 x1865013828598016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628097.826665:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828598016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628097.826746:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628097.826751:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628097.826755:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628098.017800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628098.017892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628098.785784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628099.041766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628099.041866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628099.809733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628100.065722:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628100.065795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628100.833794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628101.089780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628101.089938:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628101.857781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628102.113782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628102.113951:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628102.881770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628102.945782:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628102.945794:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628102.945801:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628102.945812:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12437 00000100:00080000:1.0:1778628102.945826:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628102.945836:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628102.945879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828598144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628102.945915:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628102.945920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628102.945923:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628102.945928:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12437 00000100:00080000:1.0:1778628102.945932:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628102.945937:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628102.945948:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628102.945962:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828598272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628102.945977:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628102.945981:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628102.945984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628102.945989:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12437 00000100:00080000:1.0:1778628102.945992:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628102.946004:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828598144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628102.946046:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628102.946055:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628102.946076:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828598400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628102.946090:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628102.946095:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628102.946097:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628102.946103:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12437 00000100:00080000:1.0:1778628102.946111:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628102.946116:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628102.946121:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628102.946144:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828598528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628102.946162:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598272:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628102.946168:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628102.946183:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628102.946187:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628102.946198:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12437 00000100:00080000:1.0:1778628102.946226:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628102.946236:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628102.946247:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628102.946271:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828598656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628102.946277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628102.946370:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628102.946405:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628103.137725:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628103.137827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628103.905738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628104.161744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628104.161779:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628104.929757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628105.185739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628105.185802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628105.953783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778628106.017994:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778628106.018042:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778628106.018062:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628106.018100:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628106.018111:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628106.018122:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628106.018135:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628106.018168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628102/real 1778628106] req@ffff954350194000 x1865013828598656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628106.018199:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828598656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628106.018214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628106.018221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628106.018226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628106.018235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628102/real 1778628106] req@ffff954350197480 x1865013828598400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628106.018249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828598400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628106.018255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628106.018258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628106.018260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628106.018265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628102/real 1778628106] req@ffff954350196d80 x1865013828598528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628106.018275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828598528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628106.018297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628106.018302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628106.018306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628106.018313:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628102/real 1778628106] req@ffff954350197800 x1865013828598144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628106.018330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828598144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628106.018339:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628106.018343:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628106.018347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828598144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628106.018355:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628102/real 1778628106] req@ffff954350194700 x1865013828598272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628106.018368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828598272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628106.018378:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628106.018383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628106.018386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628106.209779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628106.209917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628106.977799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628107.233787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628107.233957:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628108.001755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628108.065733:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628108.065742:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628108.065746:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628108.065753:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12442 00000100:00080000:1.0:1778628108.065763:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628108.065769:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628108.065799:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828598784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628108.065821:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628108.065824:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628108.065825:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628108.065828:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12442 00000100:00080000:1.0:1778628108.065830:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628108.065833:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628108.065839:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628108.065848:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828598912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628108.065854:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628108.065862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628108.065864:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628108.065866:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12442 00000100:00080000:1.0:1778628108.065868:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628108.065870:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628108.065882:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628108.065891:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828599040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628108.065899:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828598784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628108.065900:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628108.065921:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628108.065923:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628108.065927:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12442 00000100:00080000:1.0:1778628108.065930:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628108.065934:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628108.065938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628108.065949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828599168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628108.065955:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628108.065958:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628108.065959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628108.065961:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12442 00000100:00080000:1.0:1778628108.065964:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628108.065966:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628108.065970:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778628108.065973:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628108.065980:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828599296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628108.066025:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628108/real 1778628108] req@ffff954350197100 x1865013828598784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628108.066059:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828598784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628108.066070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628108.066075:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628108.066079:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828598784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628108.066089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628108.066115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628108.066129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628108/real 1778628108] req@ffff954350194700 x1865013828598912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628108.066143:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828598912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628108.066152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628108.066155:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628108.066159:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828598912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628108.066185:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628108.066207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628108.066219:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628108/real 1778628108] req@ffff954350197800 x1865013828599040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628108.066232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828599040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628108.066255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628108.066258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628108.066261:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628108.066267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628108.066291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628108.066303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628108/real 1778628108] req@ffff954350196d80 x1865013828599168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628108.066316:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828599168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628108.066323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628108.066326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628108.066329:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628108.066335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628108.066355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628108.066364:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628108/real 1778628108] req@ffff954350197480 x1865013828599296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628108.066375:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828599296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628108.066382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628108.066385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628108.066402:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628108.257742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628108.257865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628109.025791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628109.281807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628109.281862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628110.049774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628110.305787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628110.305911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628111.073739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628111.329739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628111.329790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628112.097735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628112.353762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628112.353820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628113.121745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628113.185724:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628113.185731:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628113.185754:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628113.185762:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12447 00000100:00080000:1.0:1778628113.185772:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628113.185778:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628113.185839:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828599424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628113.185863:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628113.185867:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628113.185869:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628113.185873:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12447 00000100:00080000:1.0:1778628113.185877:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628113.185881:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628113.185890:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628113.185927:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828599552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628113.185933:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828599424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628113.185938:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628113.185941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628113.185943:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628113.185946:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12447 00000100:00080000:1.0:1778628113.185950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628113.185954:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628113.185958:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628113.185967:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828599680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628113.185987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628113.185991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1778628113.185992:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628113.185993:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628113.185997:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12447 00000100:00080000:1.0:1778628113.186001:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628113.186004:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628113.186009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628113.186021:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828599808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628113.186026:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628113/real 1778628113] req@ffff954350194000 x1865013828599424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628113.186030:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628113.186034:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628113.186036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628113.186040:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12447 00000100:00080000:1.0:1778628113.186044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628113.186048:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628113.186052:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628113.186058:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828599424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628113.186062:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828599936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628113.186070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628113.186074:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628113.186078:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828599424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628113.186102:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628113.186124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628113.186135:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628113/real 1778628113] req@ffff954350194380 x1865013828599552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628113.186144:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828599552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628113.186150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628113.186152:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628113.186160:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628113.186165:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628113.186180:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628113.186187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628113/real 1778628113] req@ffff954350194a80 x1865013828599680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628113.186211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828599680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628113.186216:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628113.186218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628113.186220:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628113.186224:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628113.186234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628113.186240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628113/real 1778628113] req@ffff954350196a00 x1865013828599808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628113.186247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828599808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628113.186252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628113.186254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628113.186255:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628113.186258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628113.186267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628113.186277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628113/real 1778628113] req@ffff954350194e00 x1865013828599936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628113.186285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828599936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628113.186290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628113.186292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628113.186293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828599936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628113.377738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628113.377856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628114.145838:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628114.401744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628114.401875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628115.169749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628115.425833:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628115.425957:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628116.193898:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628116.449794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628116.449889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628117.217838:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628117.473815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628117.473875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628118.241816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628118.305863:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628118.305877:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628118.305884:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628118.305898:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12452 00000100:00080000:0.0:1778628118.305913:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628118.305924:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628118.305989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828600064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628118.306031:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628118.306036:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628118.306039:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628118.306045:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12452 00000100:00080000:0.0:1778628118.306051:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628118.306081:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628118.306094:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628118.306101:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828600064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628118.306134:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828600192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628118.306156:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628118.306163:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628118.306167:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628118.306174:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12452 00000100:00080000:0.0:1778628118.306183:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628118.306190:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628118.306199:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778628118.306208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628118.306222:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828600320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628118.306240:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628118.306274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628118/real 1778628118] req@ffff954342f49880 x1865013828600064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628118.306278:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628118.306285:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628118.306299:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12452 00000100:00080000:0.0:1778628118.306306:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778628118.306307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828600064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628118.306318:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778628118.306326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778628118.306346:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:2.0:1778628118.306359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628118.306364:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828600064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628118.306375:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828600448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628118.306396:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600192:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778628118.306397:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628118.306415:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628118.306419:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778628118.306428:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628118.306432:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12452 00000100:00080000:0.0:1778628118.306443:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778628118.306449:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628118/real 1778628118] req@ffff954342f4bb80 x1865013828600192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628118.306458:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628118.306474:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828600192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778628118.306475:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778628118.306493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628118.306497:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628118.306511:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828600576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628118.306533:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628118.306548:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628118.306574:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628118.306585:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628118/real 1778628118] req@ffff954342f48e00 x1865013828600320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628118.306597:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828600320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628118.306606:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628118.306609:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628118.306612:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628118.306626:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628118.306641:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628118.306648:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628118/real 1778628118] req@ffff954342f4aa00 x1865013828600448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628118.306658:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828600448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628118.306665:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628118.306668:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628118.306670:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628118.306675:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628118.306741:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628118.306787:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628118/real 1778628118] req@ffff954342f49180 x1865013828600576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628118.306807:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828600576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628118.306821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628118.306827:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628118.306832:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628118.497900:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628118.497941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628119.265797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628119.521779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628119.521827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628120.289793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628120.545847:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628120.545880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628121.313765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628121.569790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628121.570001:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628122.337733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628122.593746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628122.593772:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628123.361773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628123.425810:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628123.425821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628123.425827:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628123.425840:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12457 00000100:00080000:3.0:1778628123.425854:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628123.425863:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628123.425908:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828600704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628123.425945:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628123.425950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628123.425961:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628123.425970:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12457 00000100:00080000:3.0:1778628123.425977:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628123.425985:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628123.426021:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628123.426024:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828600704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628123.426054:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828600832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628123.426080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628123.426087:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628123.426090:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628123.426095:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12457 00000100:00080000:3.0:1778628123.426103:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628123.426109:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628123.426116:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778628123.426129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628123.426137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828600960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628123.426150:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628123.426155:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628123.426158:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628123.426164:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12457 00000100:00080000:3.0:1778628123.426168:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628123.426173:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628123.426179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628123.426190:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828601088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628123.426201:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628123.426205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778628123.426215:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628123/real 1778628123] req@ffff95434701d880 x1865013828600704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628123.426218:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628123.426232:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12457 00000100:00080000:3.0:1778628123.426242:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628123.426255:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628123.426263:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628123.426273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828600704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628123.426283:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828601216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628123.426298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628123.426310:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628123.426330:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828600704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628123.426377:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628123.426413:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628123.426430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628123/real 1778628123] req@ffff95434701ea00 x1865013828600832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628123.426447:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828600832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628123.426458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628123.426462:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628123.426467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628123.426474:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628123.426494:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628123.426504:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628123/real 1778628123] req@ffff95434701c000 x1865013828600960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628123.426530:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828600960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628123.426540:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628123.426544:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628123.426548:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828600960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628123.426555:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628123.426574:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628123.426598:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628123/real 1778628123] req@ffff95434701c700 x1865013828601088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628123.426641:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828601088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628123.426652:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628123.426656:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628123.426660:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628123.426667:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628123.426733:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628123.426747:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628123/real 1778628123] req@ffff95434701f800 x1865013828601216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628123.426762:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828601216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628123.426772:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628123.426776:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628123.426780:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628123.617791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628123.617849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628124.385786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628124.641786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628124.641821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628125.409833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628125.665757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628125.665880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628126.433763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628126.689871:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628126.689934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628127.457748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628127.713783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628127.713861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628128.481758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628128.545821:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628128.545829:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628128.545833:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628128.545842:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12463 00000100:00080000:1.0:1778628128.545879:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628128.545884:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628128.545917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828601344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628128.545937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628128.545940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628128.545941:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628128.545944:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12463 00000100:00080000:1.0:1778628128.545946:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628128.545949:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628128.545956:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628128.545964:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828601472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628128.545973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628128.545977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778628128.545978:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828601344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628128.545979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628128.545987:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12463 00000100:00080000:1.0:1778628128.545989:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628128.545991:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628128.545995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628128.546008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828601600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628128.546019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628128.546029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628128.546031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628128.546036:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12463 00000800:00000100:2.0:1778628128.546040:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628128.546045:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628128.546052:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628128.546058:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628128.546074:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628128/real 1778628128] req@ffff954350195500 x1865013828601344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628128.546075:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828601728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628128.546097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628128.546098:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828601344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628128.546101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628128.546105:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628128.546112:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12463 00000100:00080000:1.0:1778628128.546119:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628128.546121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628128.546123:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628128.546130:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778628128.546131:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628128.546139:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828601344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628128.546144:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828601856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628128.546164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628128.546184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628128.546192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628128/real 1778628128] req@ffff954350195880 x1865013828601472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628128.546201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828601472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628128.546206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628128.546209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628128.546211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628128.546215:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628128.546225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628128.546238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628128/real 1778628128] req@ffff954350197b80 x1865013828601600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628128.546246:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828601600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628128.546251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628128.546253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628128.546255:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628128.546259:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628128.546270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628128.546275:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628128/real 1778628128] req@ffff954350195180 x1865013828601728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628128.546282:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828601728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628128.546287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628128.546288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628128.546290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628128.546294:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628128.546303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628128.546308:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628128/real 1778628128] req@ffff954350195c00 x1865013828601856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628128.546315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828601856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628128.546324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628128.546326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628128.546328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828601856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628128.737762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628128.737788:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628129.505771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628129.761803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628129.761856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628130.529742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628130.785767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628130.785819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628131.553746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628131.809747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628131.809786:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628132.577764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628132.833748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628132.833818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628133.601744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628133.665793:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628133.665803:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628133.665808:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628133.665819:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12468 00000100:00080000:1.0:1778628133.665830:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628133.665837:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628133.665885:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828601984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628133.665915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628133.665920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628133.665922:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628133.665926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12468 00000100:00080000:1.0:1778628133.665930:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628133.665934:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628133.665943:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628133.665956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828602112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628133.665967:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628133.665971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628133.665973:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628133.665977:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12468 00000100:00080000:1.0:1778628133.665981:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628133.665984:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628133.665990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628133.665991:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828601984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628133.666003:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828602240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628133.666015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628133.666018:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628133.666020:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628133.666031:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12468 00000100:00080000:1.0:1778628133.666036:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628133.666041:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628133.666047:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778628133.666057:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628133.666063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828602368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628133.666075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628133.666078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628133.666080:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628133.666085:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12468 00000100:00080000:1.0:1778628133.666089:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628133.666093:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628133.666099:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628133/real 1778628133] req@ffff954350195f80 x1865013828601984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628133.666100:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628133.666115:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828602496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628133.666127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828601984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628133.666136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628133.666140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628133.666143:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828601984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628133.666158:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628133.666176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628133.666185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628133/real 1778628133] req@ffff954350195c00 x1865013828602112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628133.666193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828602112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628133.666199:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628133.666201:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628133.666203:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628133.666221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628133.666231:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628133.666236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628133/real 1778628133] req@ffff954350195180 x1865013828602240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628133.666243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828602240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628133.666248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628133.666250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628133.666252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628133.666255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628133.666265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628133.666270:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628133/real 1778628133] req@ffff954350197b80 x1865013828602368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628133.666283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828602368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628133.666288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628133.666290:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628133.666292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628133.666295:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628133.666306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628133.666312:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628133/real 1778628133] req@ffff954350195880 x1865013828602496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628133.666319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828602496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628133.666324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628133.666326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628133.666327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628133.857756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628133.857815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628134.625785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628134.881797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628134.881975:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628135.649726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628135.905751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628135.905886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628136.673748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628136.929814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628136.929858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628137.697723:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628137.953807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628137.954147:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628138.721726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628138.785767:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628138.785774:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628138.785778:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628138.785786:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12473 00000100:00080000:1.0:1778628138.785796:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628138.785835:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628138.785889:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828602624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628138.785913:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628138.785916:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628138.785917:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628138.785920:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12473 00000100:00080000:1.0:1778628138.785924:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628138.785927:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628138.785935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628138.785944:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828602752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628138.785950:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628138.785952:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628138.785953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628138.785955:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12473 00000100:00080000:1.0:1778628138.785957:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628138.785960:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628138.785963:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628138.785969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828602880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628138.785977:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628138.785979:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628138.785980:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628138.785986:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828602624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628138.785991:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12473 00000100:00080000:1.0:1778628138.785995:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628138.785998:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628138.786002:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628138.786012:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828603008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628138.786019:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628138.786021:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628138.786022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628138.786025:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12473 00000100:00080000:1.0:1778628138.786027:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628138.786029:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628138.786031:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628138.786039:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828603136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628138.786056:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628138.786086:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628138/real 1778628138] req@ffff954350195500 x1865013828602624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628138.786101:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828602624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628138.786112:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628138.786117:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628138.786121:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828602624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628138.786128:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628138.786150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628138.786162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628138/real 1778628138] req@ffff954350194e00 x1865013828602752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628138.786176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828602752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628138.786182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628138.786184:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628138.786187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628138.786206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628138.786220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628138.786226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628138/real 1778628138] req@ffff954350196a00 x1865013828602880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628138.786241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828602880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628138.786246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628138.786249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628138.786250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828602880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628138.786254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628138.786266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628138.786271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628138/real 1778628138] req@ffff954350194a80 x1865013828603008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628138.786278:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828603008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628138.786283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628138.786285:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628138.786287:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628138.786290:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628138.786299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628138.786323:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628138/real 1778628138] req@ffff954350194380 x1865013828603136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628138.786331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828603136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628138.786345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628138.786348:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628138.786350:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628138.977752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628138.977937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628139.745728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628140.001812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628140.001884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628140.769728:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628141.025766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628141.025874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628141.793739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628142.049754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628142.049837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628142.817774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628143.073777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628143.073888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628143.841779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628143.905749:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628143.905763:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628143.905770:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628143.905780:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12478 00000100:00080000:1.0:1778628143.905792:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628143.905798:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628143.905841:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828603264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628143.905870:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628143.905893:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628143.905895:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628143.905899:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12478 00000100:00080000:1.0:1778628143.905902:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628143.905906:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628143.905917:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628143.905928:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828603392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628143.905937:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628143.905939:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628143.905941:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628143.905947:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12478 00000100:00080000:1.0:1778628143.905953:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778628143.905954:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828603264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628143.905957:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628143.905966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628143.905974:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828603520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628143.905982:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628143.905985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628143.905987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628143.905991:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12478 00000100:00080000:1.0:1778628143.905994:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628143.905998:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628143.906008:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628143.906020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828603648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628143.906027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628143.906033:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1778628143.906034:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628143.906036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628143.906040:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12478 00000100:00080000:1.0:1778628143.906045:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628143.906048:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628143.906054:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628143.906063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828603776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628143.906079:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628143/real 1778628143] req@ffff954350194000 x1865013828603264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628143.906097:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828603264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628143.906106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628143.906109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628143.906112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828603264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628143.906132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628143.906147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628143.906171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628143/real 1778628143] req@ffff954350197480 x1865013828603392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628143.906179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828603392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628143.906185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628143.906188:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628143.906190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628143.906194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628143.906225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628143.906233:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628143/real 1778628143] req@ffff954350196d80 x1865013828603520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628143.906242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828603520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628143.906247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628143.906249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628143.906251:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628143.906255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628143.906272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628143.906278:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628143/real 1778628143] req@ffff954350197800 x1865013828603648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628143.906286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828603648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628143.906295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628143.906298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628143.906300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628143.906304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628143.906318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628143.906324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628143/real 1778628143] req@ffff954350194700 x1865013828603776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628143.906331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828603776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628143.906336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628143.906338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628143.906340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828603776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628144.097770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628144.097867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628144.865788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628145.121779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628145.121830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628145.889747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628146.145739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628146.145816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628146.913743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628147.169731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628147.169782:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628147.937752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628148.193757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628148.193803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628148.961758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628149.025731:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628149.025739:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628149.025743:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628149.025751:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12483 00000100:00080000:1.0:1778628149.025760:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628149.025766:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628149.025798:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828603904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628149.025820:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628149.025823:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628149.025824:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628149.025827:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12483 00000100:00080000:1.0:1778628149.025843:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628149.025847:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628149.025855:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628149.025860:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828603904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628149.025870:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828604032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628149.025884:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628149.025887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628149.025888:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628149.025891:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12483 00000100:00080000:1.0:1778628149.025893:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628149.025895:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628149.025898:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628149.025907:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828604160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628149.025913:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628149.025915:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628149.025917:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628149.025920:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12483 00000100:00080000:1.0:1778628149.025924:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628149.025926:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628149.025929:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778628149.025934:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628149.025939:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828604288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628149.025956:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628149.025959:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628149.025960:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628149.025964:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12483 00000100:00100000:2.0:1778628149.025966:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628149/real 1778628149] req@ffff954350197100 x1865013828603904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628149.025968:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628149.025976:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628149.025983:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628149.025991:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828603904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628149.025994:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828604416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628149.026019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628149.026026:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628149.026042:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828603904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628149.026068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628149.026090:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628149.026116:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628149/real 1778628149] req@ffff954350194700 x1865013828604032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628149.026125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828604032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628149.026131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628149.026133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628149.026136:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628149.026140:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628149.026154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628149.026160:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628149/real 1778628149] req@ffff954350197800 x1865013828604160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628149.026179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828604160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628149.026184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628149.026186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628149.026188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628149.026192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628149.026204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628149.026210:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628149/real 1778628149] req@ffff954350196d80 x1865013828604288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628149.026217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828604288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628149.026222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628149.026224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628149.026225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628149.026238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628149.026256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628149.026267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628149/real 1778628149] req@ffff954350197480 x1865013828604416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628149.026279:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828604416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628149.026299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628149.026301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628149.026304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628149.217730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628149.217811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628149.985753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628150.241763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628150.241789:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628151.009732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628151.265748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628151.265756:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628152.033748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628152.289763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628152.289798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628153.057743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628153.313768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628153.313828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628154.081761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628154.145778:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628154.145786:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628154.145790:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628154.145798:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12488 00000100:00080000:2.0:1778628154.145808:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628154.145830:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628154.145864:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828604544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628154.145895:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628154.145899:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628154.145901:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628154.145906:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12488 00000100:00080000:2.0:1778628154.145926:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628154.145930:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628154.145938:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628154.145951:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828604544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628154.145951:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828604672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628154.145975:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628154.145977:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628154.145978:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628154.145981:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12488 00000100:00080000:2.0:1778628154.145984:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628154.145986:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628154.145989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628154.146011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828604800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628154.146021:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628154.146025:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628154.146028:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628154.146032:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12488 00000100:00080000:2.0:1778628154.146036:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628154.146041:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628154.146047:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628154.146065:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628154.146066:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828604928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628154.146107:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628154/real 1778628154] req@ffff954358789500 x1865013828604544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628154.146120:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628154.146127:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628154.146129:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628154.146136:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12488 00000100:00080000:2.0:1778628154.146141:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628154.146142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828604544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628154.146151:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628154.146168:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778628154.146173:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628154.146179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628154.146184:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828605056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628154.146188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828604544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628154.146233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628154.146252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628154.146261:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628154/real 1778628154] req@ffff954358788700 x1865013828604672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628154.146270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828604672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628154.146276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628154.146278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628154.146281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628154.146284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628154.146294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628154.146300:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628154/real 1778628154] req@ffff95435878b100 x1865013828604800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628154.146325:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828604800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628154.146331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628154.146333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628154.146335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628154.146339:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628154.146353:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628154.146359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628154/real 1778628154] req@ffff95435878bb80 x1865013828604928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628154.146366:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828604928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628154.146372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628154.146373:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628154.146375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828604928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628154.146379:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628154.146388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628154.146394:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628154/real 1778628154] req@ffff95435878b480 x1865013828605056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628154.146401:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828605056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628154.146415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628154.146418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628154.146420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628154.337749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628154.337790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628155.105740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778628155.361802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628155.361802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628156.129730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628156.385764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628156.385798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628157.153743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628157.409765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628157.409770:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628158.177743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628158.433770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628158.433813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628159.201756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628159.265802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628159.265813:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628159.265818:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628159.265828:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12493 00000100:00080000:1.0:1778628159.265840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628159.265847:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628159.265884:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828605184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628159.265925:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628159.265930:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628159.265932:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628159.265937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12493 00000100:00080000:1.0:1778628159.265941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628159.265953:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628159.265972:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828605184:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778628159.265978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628159.265995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828605312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628159.266009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628159.266013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628159.266014:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628159.266019:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12493 00000100:00080000:1.0:1778628159.266024:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628159.266029:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628159.266035:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778628159.266038:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628159.266052:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828605440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628159.266064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628159.266067:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628159.266070:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628159.266075:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628159/real 1778628159] req@ffff954350194000 x1865013828605184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628159.266076:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12493 00000100:00080000:1.0:1778628159.266084:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628159.266091:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628159.266098:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828605184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628159.266098:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628159.266115:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828605568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628159.266119:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628159.266128:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778628159.266133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628159.266134:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828605184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628159.266142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628159.266151:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628159.266156:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12493 00000100:00100000:0.0:1778628159.266160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605312:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628159.266161:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628159.266169:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628159.266176:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778628159.266183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628159.266193:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828605696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628159.266196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628159/real 1778628159] req@ffff954350194380 x1865013828605312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628159.266222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828605312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628159.266240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628159.266243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628159.266245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628159.266249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628159.266263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628159.266270:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628159/real 1778628159] req@ffff954350194a80 x1865013828605440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628159.266277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828605440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628159.266282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628159.266284:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628159.266286:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628159.266290:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628159.266300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628159.266305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628159/real 1778628159] req@ffff954350196a00 x1865013828605568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628159.266312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828605568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628159.266317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628159.266319:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628159.266325:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628159.266336:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628159.266348:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628159.266354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628159/real 1778628159] req@ffff954350194e00 x1865013828605696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628159.266361:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828605696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628159.266367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628159.266369:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628159.266371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628159.457751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628159.457800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628160.225746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628160.481740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628160.481796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628161.249741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628161.505735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628161.505808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628162.273735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628162.529741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628162.529820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628163.297756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628163.553738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628163.553766:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628164.321763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628164.385737:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628164.385747:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628164.385751:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628164.385762:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12498 00000100:00080000:1.0:1778628164.385773:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628164.385779:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628164.385835:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828605824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628164.385864:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628164.385867:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628164.385869:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628164.385873:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12498 00000100:00080000:1.0:1778628164.385875:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628164.385877:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628164.385886:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628164.385914:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828605824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628164.385918:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828605952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628164.385928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628164.385931:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628164.385932:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628164.385935:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12498 00000100:00080000:1.0:1778628164.385939:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628164.385941:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628164.385945:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628164.385954:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828606080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628164.385961:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628164.385964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628164.385965:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628164.385967:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12498 00000100:00080000:1.0:1778628164.385977:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628164.385979:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628164.385983:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628164.385988:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628164.385993:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828606208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628164.386004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628164.386007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628164.386008:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628164.386016:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12498 00000100:00080000:1.0:1778628164.386020:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628164.386023:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628164.386026:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628164/real 1778628164] req@ffff954350195500 x1865013828605824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628164.386030:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628164.386043:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828606336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628164.386049:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828605824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628164.386062:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628164.386067:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628164.386072:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828605824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628164.386093:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628164.386117:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628164.386126:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628164/real 1778628164] req@ffff954350195880 x1865013828605952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628164.386135:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828605952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628164.386141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628164.386144:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628164.386146:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828605952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628164.386149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628164.386159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628164.386165:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628164/real 1778628164] req@ffff954350197b80 x1865013828606080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628164.386172:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828606080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628164.386178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628164.386179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628164.386181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628164.386185:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628164.386196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628164.386201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628164/real 1778628164] req@ffff954350195180 x1865013828606208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628164.386219:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828606208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628164.386225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628164.386227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628164.386229:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628164.386232:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628164.386245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628164.386250:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628164/real 1778628164] req@ffff954350195c00 x1865013828606336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628164.386257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828606336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628164.386263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628164.386265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628164.386267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628164.577737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628164.577779:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628165.345817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628165.601832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628165.601885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628166.369792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628166.625784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628166.625828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628167.393756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628167.649795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628167.649823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628168.417756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628168.673833:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628168.673881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628169.441858:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628169.505730:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628169.505738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628169.505741:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628169.505749:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12504 00000100:00080000:1.0:1778628169.505759:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628169.505764:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628169.505796:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828606464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628169.505818:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628169.505822:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628169.505824:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628169.505828:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12504 00000100:00080000:1.0:1778628169.505832:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628169.505835:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628169.505845:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628169.505861:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828606592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628169.505865:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828606464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628169.505882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628169.505895:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628169.505897:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628169.505901:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12504 00000100:00080000:1.0:1778628169.505920:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628169.505925:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628169.505930:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628169.505945:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828606720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628169.505956:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628169.505961:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606592:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628169.505962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628169.505966:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628169.505973:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12504 00000100:00080000:1.0:1778628169.505979:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628169.505983:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628169.505989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628169.506008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828606848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628169.506010:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606720:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628169.506025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628169.506028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628169.506029:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628169.506033:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12504 00000100:00100000:0.0:1778628169.506037:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606848:192.168.204.102@tcp:8:kworker.0 00000100:00080000:1.0:1778628169.506037:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628169.506043:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628169.506054:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628169.506064:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828606976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628169.506092:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628169.697828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628169.697916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628170.465805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628170.721811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628170.721846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628171.489775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628171.745794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628171.745817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628172.513806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778628172.578007:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778628172.578049:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778628172.578066:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628172.578111:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628172.578119:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628172.578128:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628172.578135:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628172.578183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628169/real 1778628172] req@ffff954350195880 x1865013828606976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628172.578227:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828606976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628172.578246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628172.578254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628172.578258:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628172.578267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628169/real 1778628172] req@ffff954350197b80 x1865013828606848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628172.578282:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828606848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628172.578294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628172.578298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628172.578303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628172.578311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628169/real 1778628172] req@ffff954350195c00 x1865013828606592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628172.578346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828606592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628172.578357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628172.578363:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628172.578367:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628172.578376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628169/real 1778628172] req@ffff954350195180 x1865013828606720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628172.578394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828606720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628172.578404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628172.578409:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628172.578414:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828606720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628172.578422:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628169/real 1778628172] req@ffff954350195f80 x1865013828606464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628172.578441:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828606464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628172.578452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628172.578456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628172.578460:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828606464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628172.769820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628172.769904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628173.537820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628173.793844:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628173.793903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628174.561816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628174.625829:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628174.625841:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628174.625864:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628174.625879:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12509 00000100:00080000:0.0:1778628174.625896:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628174.625906:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628174.625966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828607104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628174.626021:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628174.626027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628174.626029:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628174.626035:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12509 00000100:00080000:0.0:1778628174.626040:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628174.626063:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628174.626076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628174.626093:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828607232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628174.626106:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628174.626110:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628174.626112:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628174.626115:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12509 00000100:00080000:0.0:1778628174.626119:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628174.626127:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628174.626127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828607104:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778628174.626159:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628174.626177:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828607360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628174.626188:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628174.626192:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628174.626194:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628174.626199:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12509 00000100:00080000:0.0:1778628174.626203:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628174.626207:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628174.626212:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628174.626223:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828607488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628174.626237:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628174.626242:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628174.626245:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628174.626254:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12509 00000800:00000100:2.0:1778628174.626257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628174.626262:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628174.626271:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628174.626280:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628174.626306:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828607616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628174.626333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628174/real 1778628174] req@ffff954342f48000 x1865013828607104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628174.626392:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828607104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628174.626421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628174.626444:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628174.626451:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828607104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628174.626497:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628174.626543:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628174.626567:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628174/real 1778628174] req@ffff954342f48380 x1865013828607232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628174.626588:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828607232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628174.626627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628174.626634:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628174.626641:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628174.626652:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628174.626753:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628174.626775:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628174/real 1778628174] req@ffff954342f49180 x1865013828607360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628174.626798:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828607360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628174.626812:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628174.626818:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628174.626824:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628174.626835:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628174.626898:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628174.626921:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628174/real 1778628174] req@ffff954342f4aa00 x1865013828607488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628174.626942:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828607488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628174.626957:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628174.626964:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628174.626969:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628174.626980:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628174.627014:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628174.627034:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628174/real 1778628174] req@ffff954342f48e00 x1865013828607616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628174.627053:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828607616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628174.627066:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628174.627071:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628174.627076:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628174.817804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628174.817853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628175.585781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778628175.841934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778628175.841963:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628176.609823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628176.865739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628176.865828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628177.633759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628177.889787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628177.889832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628178.657822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628178.913781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628178.913848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628179.681798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628179.745829:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628179.745847:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628179.745856:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628179.745875:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12514 00000100:00080000:0.0:1778628179.745894:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628179.745904:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628179.745963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828607744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628179.746008:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628179.746016:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628179.746019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628179.746029:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12514 00000100:00080000:0.0:1778628179.746036:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628179.746043:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628179.746064:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628179.746093:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828607872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628179.746099:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828607744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628179.746125:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628179.746131:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628179.746133:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628179.746142:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12514 00000100:00080000:0.0:1778628179.746148:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628179.746154:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628179.746181:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628179.746201:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828608000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628179.746216:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778628179.746223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628179.746225:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628179.746230:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628179.746238:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12514 00000100:00080000:0.0:1778628179.746247:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628179.746254:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628179.746263:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628179.746282:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828608128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628179.746292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628179/real 1778628179] req@ffff954342f4bb80 x1865013828607744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628179.746302:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628179.746318:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628179.746325:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628179.746334:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12514 00000100:00080000:0.0:1778628179.746340:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778628179.746349:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828607744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628179.746350:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628179.746363:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778628179.746382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778628179.746389:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828608256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628179.746397:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628179.746412:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828607744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628179.746454:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628179.746484:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628179.746529:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628179/real 1778628179] req@ffff954342f49880 x1865013828607872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628179.746548:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828607872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628179.746559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628179.746563:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628179.746568:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828607872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628179.746576:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628179.746601:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628179.746612:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628179/real 1778628179] req@ffff954342f4a300 x1865013828608000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628179.746650:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828608000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628179.746660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628179.746665:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628179.746711:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628179.746721:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628179.746754:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628179.746776:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628179/real 1778628179] req@ffff954342f4b480 x1865013828608128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628179.746800:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828608128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628179.746816:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628179.746823:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628179.746830:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628179.746842:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628179.746882:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628179.746902:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628179/real 1778628179] req@ffff954342f49c00 x1865013828608256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628179.746918:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828608256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628179.746947:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628179.746951:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628179.746956:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628179.937825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628179.937968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628180.705857:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628180.961786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628180.961953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628181.729875:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628181.985756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628181.985879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628182.753795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628183.009725:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628183.009814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628183.777903:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628184.033771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628184.033897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628184.801765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628184.865779:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628184.865789:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628184.865794:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628184.865806:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12519 00000100:00080000:2.0:1778628184.865820:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628184.865826:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628184.865870:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828608384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628184.865897:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628184.865900:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628184.865902:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628184.865905:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12519 00000100:00080000:2.0:1778628184.865908:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628184.865911:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628184.865921:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628184.865931:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828608512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628184.865937:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828608384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628184.865962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628184.865964:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628184.865966:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628184.865969:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12519 00000100:00080000:2.0:1778628184.865972:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628184.865975:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628184.865979:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628184.865991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828608640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628184.865998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628184.866000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628184.866002:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628184.866004:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12519 00000100:00080000:2.0:1778628184.866007:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628184.866009:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628184.866012:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628184.866019:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828608768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628184.866028:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778628184.866029:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628184.866032:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628184.866034:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628184.866038:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12519 00000100:00080000:2.0:1778628184.866045:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628184.866049:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628184.866053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628184.866062:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828608896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628184.866080:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628184/real 1778628184] req@ffff954358788380 x1865013828608384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628184.866096:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828608384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628184.866105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628184.866108:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628184.866112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828608384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628184.866135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628184.866161:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628184.866171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628184/real 1778628184] req@ffff95435878aa00 x1865013828608512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628184.866180:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828608512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628184.866186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628184.866188:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628184.866191:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628184.866194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628184.866206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628184.866212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628184/real 1778628184] req@ffff95435878b480 x1865013828608640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628184.866230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828608640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628184.866236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628184.866239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628184.866242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628184.866249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628184.866273:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628184.866286:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628184/real 1778628184] req@ffff95435878bb80 x1865013828608768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628184.866300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828608768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628184.866309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628184.866313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628184.866317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628184.866324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628184.866346:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628184.866358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628184/real 1778628184] req@ffff95435878b100 x1865013828608896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628184.866366:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828608896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628184.866382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628184.866385:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628184.866387:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828608896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628185.057773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628185.057848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628185.825843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628186.081812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628186.081990:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628186.849866:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628187.105790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628187.105973:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628187.873748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628188.129810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628188.129920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628188.897801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628189.153767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628189.153870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628189.921833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628189.985827:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628189.985840:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628189.985846:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628189.985858:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12524 00000100:00080000:1.0:1778628189.985890:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628189.985897:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628189.985937:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828609024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628189.985966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628189.985970:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628189.985971:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628189.985975:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12524 00000100:00080000:1.0:1778628189.985979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628189.985997:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628189.986007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628189.986020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828609152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628189.986032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628189.986036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628189.986039:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628189.986043:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12524 00000100:00080000:1.0:1778628189.986047:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628189.986049:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828609024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628189.986053:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628189.986059:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628189.986070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828609280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628189.986080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628189.986083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628189.986085:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628189.986089:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12524 00000100:00080000:1.0:1778628189.986104:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628189.986108:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628189.986113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778628189.986113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628189.986127:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828609408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628189.986137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628189.986145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628189.986148:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628189.986149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628189/real 1778628189] req@ffff954350195500 x1865013828609024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628189.986155:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12524 00000100:00080000:1.0:1778628189.986159:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628189.986167:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628189.986173:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628189.986177:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828609024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628189.986186:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828609536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628189.986187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628189.986195:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628189.986203:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828609024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628189.986224:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628189.986240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628189.986248:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628189/real 1778628189] req@ffff954350194e00 x1865013828609152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628189.986256:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828609152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628189.986262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628189.986264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628189.986266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628189.986277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628189.986322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628189.986333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628189/real 1778628189] req@ffff954350196a00 x1865013828609280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628189.986344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828609280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628189.986351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628189.986353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628189.986356:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628189.986361:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628189.986376:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628189.986383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628189/real 1778628189] req@ffff954350194a80 x1865013828609408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628189.986392:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828609408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628189.986398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628189.986401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628189.986403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628189.986407:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628189.986419:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628189.986425:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628189/real 1778628189] req@ffff954350194380 x1865013828609536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628189.986441:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828609536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628189.986449:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628189.986451:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628189.986454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628190.177761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628190.177831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628190.945790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628191.201786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628191.201841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628191.969776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628192.225788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628192.225829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628192.993806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628193.249752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628193.249816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628194.017753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628194.273756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628194.273808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628195.041784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628195.105866:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628195.105882:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628195.105890:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628195.105907:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12529 00000100:00080000:1.0:1778628195.105926:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628195.105938:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628195.106021:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828609664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628195.106067:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628195.106075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628195.106078:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628195.106087:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12529 00000100:00080000:1.0:1778628195.106094:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628195.106100:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628195.106112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628195.106134:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828609792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628195.106152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828609664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628195.106182:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628195.106213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628195.106216:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628195.106224:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12529 00000100:00080000:1.0:1778628195.106242:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628195.106247:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628195.106256:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628195.106275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828609920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628195.106288:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628195.106292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628195.106294:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628195.106299:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12529 00000100:00080000:1.0:1778628195.106304:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628195.106312:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628195.106317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778628195.106320:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628195.106336:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828610048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628195.106351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628195.106355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628195.106357:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628195.106361:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12529 00000100:00080000:1.0:1778628195.106376:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628195.106382:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628195.106390:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628195.106398:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628195/real 1778628195] req@ffff954350194000 x1865013828609664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628195.106419:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828610176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628195.106432:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828609664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628195.106460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628195.106468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628195.106474:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828609664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628195.106508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628195.106544:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628195.106561:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628195/real 1778628195] req@ffff954350197480 x1865013828609792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628195.106601:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828609792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628195.106612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628195.106617:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628195.106621:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628195.106644:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628195.106724:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628195.106743:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628195/real 1778628195] req@ffff954350196d80 x1865013828609920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628195.106759:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828609920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628195.106770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628195.106774:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628195.106778:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828609920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628195.106786:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628195.106809:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628195.106820:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628195/real 1778628195] req@ffff954350197800 x1865013828610048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628195.106855:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828610048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628195.106884:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628195.106889:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628195.106893:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628195.106900:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628195.106926:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628195.106939:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628195/real 1778628195] req@ffff954350194700 x1865013828610176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628195.106952:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828610176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628195.106963:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628195.106967:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628195.106971:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628195.297789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628195.297826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628196.065809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628196.321816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628196.321836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628197.089783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628197.345809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628197.345888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628198.113824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628198.369777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628198.369814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628199.137748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628199.393735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628199.393777:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628200.161747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628200.225870:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628200.225882:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628200.225888:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628200.225899:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12534 00000100:00080000:0.0:1778628200.225913:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628200.225923:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628200.226001:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828610304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628200.226059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628200.226064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628200.226066:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628200.226072:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12534 00000100:00080000:0.0:1778628200.226078:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628200.226082:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628200.226118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628200.226143:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828610432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628200.226157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628200.226162:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628200.226165:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628200.226170:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12534 00000100:00080000:0.0:1778628200.226175:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628200.226179:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628200.226185:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628200.226196:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828610560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628200.226210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628200.226215:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828610304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628200.226216:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628200.226222:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628200.226240:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12534 00000100:00080000:0.0:1778628200.226245:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628200.226251:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628200.226260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628200.226299:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828610688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628200.226320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778628200.226325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628200.226334:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628200.226339:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628200.226350:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12534 00000100:00080000:0.0:1778628200.226360:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628200.226370:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628200.226381:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628200.226383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628200/real 1778628200] req@ffff954342f48700 x1865013828610304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628200.226416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828610816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628200.226426:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828610304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628200.226466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628200.226491:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628200.226501:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828610304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628200.226517:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628200.226565:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628200.226594:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628200/real 1778628200] req@ffff954342f4b100 x1865013828610432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628200.226612:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828610432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628200.226623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628200.226627:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628200.226631:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628200.226662:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628200.226733:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628200.226748:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628200/real 1778628200] req@ffff954342f49f80 x1865013828610560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628200.226763:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828610560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628200.226773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628200.226777:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628200.226781:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628200.226788:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628200.226827:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628200.226838:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628200/real 1778628200] req@ffff954342f49500 x1865013828610688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628200.226853:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828610688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628200.226863:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628200.226867:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628200.226871:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628200.226877:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628200.226894:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628200.226904:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628200/real 1778628200] req@ffff954342f4ad80 x1865013828610816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628200.226917:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828610816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628200.226927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628200.226931:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628200.226934:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828610816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628200.417787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628200.417967:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628201.185778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628201.441892:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628201.442044:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628202.209744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628202.465745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628202.466076:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628203.233764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628203.489871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628203.490116:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628204.257812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628204.514053:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628204.514227:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628205.281821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628205.345774:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628205.345788:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628205.345795:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628205.345809:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12539 00000100:00080000:0.0:1778628205.345825:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628205.345851:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628205.345927:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828610944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628205.345974:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628205.346039:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628205.346042:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628205.346048:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12539 00000100:00080000:0.0:1778628205.346054:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628205.346059:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628205.346072:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628205.346089:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828611072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628205.346101:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628205.346107:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628205.346109:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628205.346114:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12539 00000100:00080000:0.0:1778628205.346136:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628205.346146:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628205.346148:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828610944:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778628205.346154:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628205.346190:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828611200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628205.346242:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628205.346248:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628205.346252:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628205.346261:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12539 00000100:00080000:0.0:1778628205.346269:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778628205.346271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628205.346285:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628205.346306:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628205.346328:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628205/real 1778628205] req@ffff954342f48a80 x1865013828610944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628205.346337:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828611328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628205.346376:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828610944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628205.346378:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628205.346404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628205.346412:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628205.346421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828610944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628205.346423:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628205.346438:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628205.346445:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611072:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778628205.346463:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12539 00000100:00080000:0.0:1778628205.346477:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778628205.346491:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628205.346492:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628205.346520:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628205/real 1778628205] req@ffff954342f4a680 x1865013828611072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778628205.346523:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628205.346552:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828611072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628205.346564:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778628205.346565:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828611456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628205.346576:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628205.346589:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628205.346770:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628205.346807:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628205.346832:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628205/real 1778628205] req@ffff954342f4ad80 x1865013828611200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628205.346897:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828611200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628205.346907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628205.346911:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628205.346914:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628205.346922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628205.346946:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628205.346962:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628205/real 1778628205] req@ffff954342f49500 x1865013828611328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628205.346975:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828611328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628205.347012:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628205.347017:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628205.347020:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628205.347028:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628205.347053:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628205.347065:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628205/real 1778628205] req@ffff954342f49f80 x1865013828611456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628205.347078:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828611456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628205.347104:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628205.347108:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628205.347112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611456:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778628205.537953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628205.537972:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628206.305747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628206.561923:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778628206.562011:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628207.329808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628207.585801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628207.585915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628208.353875:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628208.609761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628208.609993:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628209.377772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628209.633757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628209.633896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628210.401916:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628210.465798:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628210.465811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628210.465818:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628210.465830:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12544 00000100:00080000:0.0:1778628210.465843:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628210.465853:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628210.465921:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828611584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628210.465958:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628210.465963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628210.465966:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628210.465972:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12544 00000100:00080000:0.0:1778628210.465994:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628210.465999:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628210.466009:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628210.466029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828611712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628210.466057:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628210.466062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628210.466065:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628210.466071:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12544 00000100:00080000:0.0:1778628210.466078:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628210.466085:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628210.466095:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628210.466097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828611584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628210.466120:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828611840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628210.466134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628210.466138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628210.466140:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628210.466156:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12544 00000100:00080000:0.0:1778628210.466160:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628210.466165:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628210.466170:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628210.466185:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828611968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628210.466206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628210.466212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628210.466214:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628210.466217:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12544 00000100:00080000:0.0:1778628210.466222:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628210.466225:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778628210.466237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778628210.466240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628210.466255:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828612096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628210.466296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628210/real 1778628210] req@ffff954342f4b100 x1865013828611584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628210.466320:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828611584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628210.466334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628210.466339:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628210.466343:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828611584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628210.466351:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628210.466374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628210.466385:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628210/real 1778628210] req@ffff954342f48700 x1865013828611712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628210.466456:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828611712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628210.466465:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628210.466469:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628210.466472:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628210.466522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628210.466543:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628210.466554:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628210/real 1778628210] req@ffff954342f49c00 x1865013828611840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628210.466565:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828611840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628210.466573:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628210.466576:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628210.466579:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628210.466584:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628210.466600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628210.466608:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628210/real 1778628210] req@ffff954342f4b480 x1865013828611968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628210.466618:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828611968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628210.466638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628210.466642:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628210.466645:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828611968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628210.466655:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628210.466682:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628210.466764:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628210/real 1778628210] req@ffff954342f4a300 x1865013828612096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628210.466782:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828612096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628210.466794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628210.466799:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628210.466804:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628210.657794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628210.657875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628211.425828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628211.681750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628211.681810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628212.449841:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628212.705743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628212.705785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628213.473851:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628213.729739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628213.729805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628214.497719:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628214.753754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628214.753821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628215.521729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628215.585799:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628215.585827:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628215.585832:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628215.585840:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12550 00000100:00080000:1.0:1778628215.585848:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628215.585856:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628215.585893:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828612224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628215.585958:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628215.585963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628215.585965:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628215.585971:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12550 00000100:00080000:1.0:1778628215.585977:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628215.585982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628215.585992:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628215.586011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828612352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628215.586041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628215.586044:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828612224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628215.586046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628215.586060:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628215.586068:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12550 00000100:00080000:1.0:1778628215.586073:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628215.586076:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628215.586082:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628215.586096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828612480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628215.586106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628215.586109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628215.586111:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628215.586116:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12550 00000100:00080000:1.0:1778628215.586128:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778628215.586128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628215.586137:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628215.586143:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628215.586158:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828612608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628215.586164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628215/real 1778628215] req@ffff954350197100 x1865013828612224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628215.586169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628215.586175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628215.586179:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628215.586185:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12550 00000100:00080000:1.0:1778628215.586189:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628215.586192:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628215.586196:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628215.586200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828612224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628215.586214:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828612736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628215.586234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628215.586239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628215.586243:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828612224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628215.586264:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628215.586283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628215.586292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628215/real 1778628215] req@ffff954350194700 x1865013828612352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628215.586300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828612352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628215.586307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628215.586311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628215.586314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628215.586320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628215.586339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628215.586350:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628215/real 1778628215] req@ffff954350197800 x1865013828612480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628215.586363:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828612480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628215.586372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628215.586375:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628215.586377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628215.586387:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628215.586406:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628215.586417:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628215/real 1778628215] req@ffff954350196d80 x1865013828612608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628215.586429:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828612608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628215.586437:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628215.586452:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628215.586456:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628215.586462:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628215.586482:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628215.586491:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628215/real 1778628215] req@ffff954350197480 x1865013828612736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628215.586499:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828612736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628215.586505:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628215.586507:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628215.586508:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628215.777740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628215.777874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628216.545788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628216.801819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628216.801903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628217.569807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628217.825782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628217.825841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628218.593739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628218.849759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628218.849836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628219.617786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628219.873777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628219.873817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628220.641765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628220.705747:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628220.705754:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628220.705758:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628220.705766:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12555 00000100:00080000:1.0:1778628220.705775:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628220.705782:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628220.705827:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828612864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628220.705856:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628220.705860:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628220.705862:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628220.705866:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12555 00000100:00080000:1.0:1778628220.705871:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628220.705875:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628220.705884:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628220.705897:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828612864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628220.705900:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828612992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628220.705927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628220.705931:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628220.705933:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628220.705937:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12555 00000100:00080000:1.0:1778628220.705942:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628220.705946:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628220.705951:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628220.705962:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828613120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628220.705963:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628220.705972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628220.705975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628220.705976:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628220.705984:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12555 00000100:00080000:1.0:1778628220.705988:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628220.705999:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628220.706001:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628220/real 1778628220] req@ffff954350194000 x1865013828612864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628220.706004:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628220.706018:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828613248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628220.706024:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828612864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628220.706028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628220.706035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628220.706038:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628220.706044:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12555 00000100:00080000:2.0:1778628220.706045:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628220.706047:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628220.706051:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778628220.706053:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628220.706058:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628220.706058:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828612864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628220.706070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828613376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628220.706087:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628220.706112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628220.706126:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628220/real 1778628220] req@ffff954350194380 x1865013828612992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628220.706138:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828612992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628220.706144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628220.706147:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628220.706149:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828612992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628220.706153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628220.706168:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628220.706174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628220/real 1778628220] req@ffff954350194a80 x1865013828613120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628220.706189:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828613120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628220.706194:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628220.706197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628220.706199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628220.706202:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628220.706225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628220.706231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628220/real 1778628220] req@ffff954350196a00 x1865013828613248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628220.706239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828613248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628220.706244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628220.706246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628220.706248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628220.706252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628220.706265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628220.706274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628220/real 1778628220] req@ffff954350194e00 x1865013828613376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628220.706285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828613376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628220.706307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628220.706311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628220.706322:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628220.897733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628220.897798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628221.665786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628221.921754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628221.921836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628222.689778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628222.945806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628222.945878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628223.713795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628223.969760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628223.969845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628224.737820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628224.993768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628224.993825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628225.761891:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628225.825828:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628225.825841:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628225.825847:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628225.825860:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12560 00000100:00080000:2.0:1778628225.825875:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628225.825884:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628225.825933:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828613504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628225.825983:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628225.825987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628225.825989:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628225.825994:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12560 00000100:00080000:2.0:1778628225.825999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628225.826003:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628225.826034:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628225.826049:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828613632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628225.826059:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828613504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628225.826066:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628225.826071:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628225.826073:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628225.826077:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12560 00000100:00080000:2.0:1778628225.826082:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628225.826086:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628225.826091:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628225.826105:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828613760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628225.826114:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628225.826118:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628225.826119:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628225.826122:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12560 00000100:00080000:2.0:1778628225.826126:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628225.826129:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628225.826133:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628225.826147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628225.826148:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828613888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628225.826161:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628225.826165:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628225.826173:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628225.826178:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12560 00000100:00080000:2.0:1778628225.826183:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628225.826206:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628225.826213:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628225.826224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628225/real 1778628225] req@ffff954358788700 x1865013828613504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628225.826229:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828614016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628225.826284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828613504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628225.826300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628225.826308:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628225.826313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828613504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628225.826344:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628225.826396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628225.826409:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628225/real 1778628225] req@ffff954358789500 x1865013828613632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628225.826433:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828613632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628225.826442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628225.826445:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628225.826448:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628225.826454:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628225.826473:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628225.826483:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628225/real 1778628225] req@ffff954358789880 x1865013828613760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628225.826494:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828613760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628225.826501:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628225.826504:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628225.826507:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628225.826512:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628225.826544:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628225.826555:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628225/real 1778628225] req@ffff954358789c00 x1865013828613888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628225.826568:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828613888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628225.826627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628225.826650:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628225.826655:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828613888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628225.826664:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628225.826740:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628225.826753:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628225/real 1778628225] req@ffff954358788e00 x1865013828614016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628225.826764:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828614016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628225.826773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628225.826776:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628225.826779:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628226.017820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628226.017931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628226.785786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628227.041780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628227.041849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628227.809818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628228.065803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628228.065979:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628228.833804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628229.089775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628229.089832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628229.857806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628230.113794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628230.113914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628230.881783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628230.945875:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628230.945886:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628230.945891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628230.945918:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12565 00000100:00080000:0.0:1778628230.945927:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628230.945934:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628230.945987:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828614144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628230.946025:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628230.946065:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628230.946068:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628230.946074:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12565 00000100:00080000:0.0:1778628230.946080:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628230.946087:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628230.946101:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628230.946118:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828614272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628230.946127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828614144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628230.946160:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628230.946164:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628230.946165:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628230.946169:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12565 00000100:00080000:0.0:1778628230.946172:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628230.946174:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628230.946178:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628230.946188:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828614400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628230.946196:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628230.946199:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628230.946200:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628230.946204:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12565 00000100:00080000:0.0:1778628230.946235:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778628230.946244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628230.946247:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628230.946265:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628230.946283:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828614528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628230.946296:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628230.946302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628230/real 1778628230] req@ffff954342f49880 x1865013828614144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628230.946303:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628230.946309:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628230.946324:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12565 00000100:00080000:0.0:1778628230.946337:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628230.946345:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628230.946351:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628230.946353:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828614144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628230.946370:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828614656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628230.946373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628230.946391:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628230.946402:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828614144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628230.946451:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628230.946525:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628230.946540:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628230/real 1778628230] req@ffff954342f4bb80 x1865013828614272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628230.946554:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828614272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628230.946561:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628230.946564:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628230.946566:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628230.946589:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628230.946614:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628230.946625:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628230/real 1778628230] req@ffff954342f48e00 x1865013828614400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628230.946638:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828614400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628230.946647:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628230.946650:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628230.946654:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628230.946660:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628230.946677:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628230.946687:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628230/real 1778628230] req@ffff954342f4aa00 x1865013828614528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628230.946741:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828614528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628230.946752:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628230.946756:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628230.946759:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628230.946766:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628230.946803:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628230.946814:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628230/real 1778628230] req@ffff954342f49180 x1865013828614656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628230.946827:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828614656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628230.946846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628230.946850:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628230.946853:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628231.137794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628231.137924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628231.905739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628232.161778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628232.161865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628232.929802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628233.185783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628233.185877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628233.953771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628234.209765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628234.209852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628234.977814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628235.233840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628235.233905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628236.001770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628236.065783:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628236.065794:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628236.065799:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628236.065809:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12570 00000100:00080000:1.0:1778628236.065821:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628236.065830:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628236.065866:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828614784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628236.065895:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628236.065900:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628236.065902:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628236.065916:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12570 00000100:00080000:1.0:1778628236.065924:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628236.065928:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628236.065937:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628236.065951:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828614912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628236.065962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628236.065966:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628236.065968:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628236.065972:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12570 00000100:00080000:1.0:1778628236.065975:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628236.065978:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628236.065991:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828614784:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778628236.065997:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628236.066011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828615040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628236.066023:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628236.066026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628236.066028:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628236.066033:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12570 00000100:00080000:1.0:1778628236.066039:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628236.066046:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628236.066053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628236.066072:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828615168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628236.066087:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628236.066094:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628236.066097:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628236.066103:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614912:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628236.066110:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12570 00000100:00080000:1.0:1778628236.066118:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628236.066124:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628236.066134:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628236.066160:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828615296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628236.066169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628236.066213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628236.066258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628236.257788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628236.257933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628237.025819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628237.281748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628237.281973:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628238.049787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628238.305774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628238.305875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628239.073782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778628239.138011:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778628239.138069:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778628239.138093:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628239.138154:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628239.138166:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628239.138175:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628239.138183:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778628239.138273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628236/real 1778628239] req@ffff954350197b80 x1865013828615296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628239.138307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828615296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628239.138349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628239.138358:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628239.138366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628239.138378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628236/real 1778628239] req@ffff954350195180 x1865013828615040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628239.138401:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828615040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628239.138430:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628239.138434:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628239.138439:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628239.138450:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628236/real 1778628239] req@ffff954350195c00 x1865013828615168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628239.138471:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828615168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628239.138484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628239.138489:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628239.138495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628239.138506:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628236/real 1778628239] req@ffff954350195500 x1865013828614784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628239.138523:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828614784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628239.138536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628239.138540:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628239.138546:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828614784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628239.138556:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628236/real 1778628239] req@ffff954350195f80 x1865013828614912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628239.138576:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828614912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628239.138600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628239.138607:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628239.138613:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828614912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628239.329775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628239.329880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628240.097852:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628240.353820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628240.353931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628241.121759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628241.185743:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628241.185752:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628241.185785:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628241.185794:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12575 00000100:00080000:1.0:1778628241.185814:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628241.185823:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628241.185860:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828615424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628241.185892:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628241.185896:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628241.185898:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628241.185903:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12575 00000100:00080000:1.0:1778628241.185908:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628241.185913:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628241.185924:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628241.185962:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828615552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628241.185970:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828615424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628241.185984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628241.185987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628241.185989:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628241.185993:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12575 00000100:00080000:1.0:1778628241.185997:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628241.186001:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628241.186006:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628241.186019:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828615680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628241.186043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778628241.186049:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628241.186050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628241.186054:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628241.186060:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12575 00000100:00080000:1.0:1778628241.186066:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628241.186071:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628241.186077:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628241.186090:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828615808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628241.186101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628241.186106:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628241/real 1778628241] req@ffff954350195880 x1865013828615424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628241.186108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628241.186111:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628241.186119:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12575 00000100:00080000:1.0:1778628241.186126:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628241.186131:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628241.186136:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628241.186142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828615424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628241.186150:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828615936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628241.186159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628241.186168:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628241.186174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828615424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628241.186212:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628241.186237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628241.186250:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628241/real 1778628241] req@ffff954350195f80 x1865013828615552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628241.186262:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828615552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628241.186305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628241.186310:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628241.186313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628241.186320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628241.186345:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628241.186356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628241/real 1778628241] req@ffff954350195500 x1865013828615680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628241.186368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828615680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628241.186377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628241.186380:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628241.186383:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628241.186390:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628241.186409:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628241.186420:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628241/real 1778628241] req@ffff954350195c00 x1865013828615808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628241.186432:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828615808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628241.186441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628241.186444:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628241.186454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628241.186461:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628241.186484:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628241.186495:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628241/real 1778628241] req@ffff954350195180 x1865013828615936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628241.186508:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828615936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628241.186517:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628241.186520:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628241.186523:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828615936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628241.377723:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628241.377799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628242.145737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628242.401735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628242.401828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628243.169734:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628243.425746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628243.425846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628244.193752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628244.449749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628244.449865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628245.217803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628245.473765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628245.473866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628246.241769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628246.305767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628246.305779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628246.305784:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628246.305795:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12580 00000100:00080000:1.0:1778628246.305832:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628246.305841:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628246.305882:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828616064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628246.305919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628246.305923:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628246.305926:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628246.305931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12580 00000100:00080000:1.0:1778628246.305934:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628246.305938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628246.305949:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628246.305963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828616192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628246.305976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628246.305981:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628246.305982:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628246.305987:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12580 00000100:00080000:1.0:1778628246.305990:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628246.305994:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628246.305999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628246.306014:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828616320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628246.306016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828616064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628246.306041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628246.306053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628246.306055:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628246.306061:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12580 00000100:00080000:1.0:1778628246.306066:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628246.306071:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628246.306078:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628246.306094:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828616448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628246.306107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628246.306111:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628246.306113:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628246.306116:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12580 00000100:00080000:1.0:1778628246.306120:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628246.306125:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628246.306135:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778628246.306135:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628246.306154:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828616576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628246.306239:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628246/real 1778628246] req@ffff954350197b80 x1865013828616064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628246.306292:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828616064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628246.306304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628246.306309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628246.306314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828616064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628246.306324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628246.306374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628246.306390:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628246/real 1778628246] req@ffff954350194e00 x1865013828616192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628246.306406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828616192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628246.306416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628246.306420:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628246.306424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628246.306455:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628246.306498:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628246.306535:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628246/real 1778628246] req@ffff954350196a00 x1865013828616320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628246.306557:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828616320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628246.306594:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628246.306602:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628246.306609:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628246.306620:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628246.306653:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628246.306667:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628246/real 1778628246] req@ffff954350194a80 x1865013828616448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628246.306681:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828616448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628246.306730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628246.306734:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628246.306738:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628246.306746:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628246.306768:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628246.306780:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628246/real 1778628246] req@ffff954350194380 x1865013828616576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628246.306793:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828616576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628246.306803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628246.306807:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628246.306819:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628246.497764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628246.497901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628247.265772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628247.521802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628247.521904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628248.289810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628248.545751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628248.545818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628249.313818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628249.569786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628249.569894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628250.337810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628250.593850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628250.593984:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628251.361825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628251.425763:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628251.425773:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628251.425784:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628251.425795:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12585 00000100:00080000:1.0:1778628251.425807:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628251.425815:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628251.425858:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828616704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628251.425918:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628251.425924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628251.425927:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628251.425934:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12585 00000100:00080000:1.0:1778628251.425941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628251.425949:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628251.425960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628251.425976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828616832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628251.425987:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828616704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628251.425992:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628251.426001:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628251.426003:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628251.426008:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12585 00000100:00080000:1.0:1778628251.426011:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628251.426016:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628251.426023:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628251.426040:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828616960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628251.426055:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628251.426061:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628251.426065:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628251.426089:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12585 00000800:00000100:2.0:1778628251.426100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628251.426111:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628251.426121:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628251.426129:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628251.426145:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828617088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628251.426150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628251/real 1778628251] req@ffff954350194000 x1865013828616704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628251.426177:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628251.426182:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628251.426184:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628251.426192:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12585 00000100:00100000:2.0:1778628251.426194:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828616704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628251.426197:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628251.426207:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628251.426217:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778628251.426220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628251.426226:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628251.426234:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828617216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628251.426235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828616704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628251.426273:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628251.426299:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628251.426311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628251/real 1778628251] req@ffff954350197480 x1865013828616832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628251.426323:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828616832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628251.426340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628251.426344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628251.426348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628251.426354:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628251.426374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628251.426383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628251/real 1778628251] req@ffff954350196d80 x1865013828616960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628251.426394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828616960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628251.426401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628251.426404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628251.426407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828616960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628251.426412:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628251.426427:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628251.426435:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628251/real 1778628251] req@ffff954350197800 x1865013828617088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628251.426446:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828617088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628251.426467:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628251.426479:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628251.426493:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628251.426504:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628251.426537:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628251.426555:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628251/real 1778628251] req@ffff954350194700 x1865013828617216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628251.426575:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828617216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628251.426587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628251.426591:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628251.426596:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628251.617783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628251.617862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628252.385756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628252.641822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628252.641901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628253.409744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628253.665781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628253.665896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628254.433890:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628254.689754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628254.689864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628255.457785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628255.713831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628255.713970:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628256.481772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628256.545835:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628256.545845:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628256.545850:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628256.545858:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12591 00000100:00080000:0.0:1778628256.545879:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628256.545886:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628256.545921:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828617344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628256.545987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628256.545990:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628256.545992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628256.545996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12591 00000100:00080000:0.0:1778628256.545999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628256.546031:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628256.546041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628256.546055:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828617472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628256.546066:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628256.546069:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628256.546071:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628256.546077:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12591 00000100:00100000:2.0:1778628256.546077:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828617344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628256.546081:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628256.546089:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628256.546093:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628256.546104:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828617600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628256.546112:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628256.546115:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628256.546117:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628256.546120:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12591 00000100:00080000:0.0:1778628256.546130:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628256.546150:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628256.546155:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628256.546166:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828617728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628256.546171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628256.546180:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628256.546183:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628256.546185:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628256.546189:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12591 00000100:00080000:0.0:1778628256.546193:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628256.546214:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628256.546219:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628256.546228:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828617856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628256.546233:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628256/real 1778628256] req@ffff954342f48380 x1865013828617344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628256.546260:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828617344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628256.546271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628256.546275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628256.546290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828617344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628256.546313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628256.546334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628256.546344:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628256/real 1778628256] req@ffff954342f48000 x1865013828617472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628256.546354:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828617472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628256.546361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628256.546364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628256.546367:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628256.546371:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628256.546383:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628256.546390:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628256/real 1778628256] req@ffff954342f49180 x1865013828617600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628256.546399:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828617600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628256.546405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628256.546408:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628256.546410:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628256.546414:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628256.546426:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628256.546441:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628256/real 1778628256] req@ffff954342f4aa00 x1865013828617728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628256.546451:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828617728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628256.546458:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628256.546461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628256.546463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628256.546468:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628256.546481:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628256.546514:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628256/real 1778628256] req@ffff954342f48e00 x1865013828617856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628256.546524:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828617856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628256.546530:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628256.546533:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628256.546535:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828617856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628256.737764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628256.737834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628257.505787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628257.761850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628257.761910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628258.529817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628258.785826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628258.785975:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628259.553801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628259.809770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628259.809838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628260.577938:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628260.833804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628260.833943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628261.601770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628261.665830:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628261.665842:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628261.665899:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628261.665910:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12596 00000100:00080000:2.0:1778628261.665921:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628261.665930:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628261.665979:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828617984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628261.666011:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628261.666015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628261.666017:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628261.666021:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12596 00000100:00080000:2.0:1778628261.666025:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628261.666029:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628261.666038:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628261.666051:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828618112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628261.666059:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828617984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628261.666067:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628261.666071:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628261.666073:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628261.666077:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12596 00000100:00080000:2.0:1778628261.666080:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628261.666083:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628261.666088:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628261.666100:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828618240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628261.666124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628261.666127:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628261.666128:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628261.666134:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12596 00000800:00000100:0.0:1778628261.666141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628261.666143:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628261.666150:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628261.666164:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628261.666177:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828618368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628261.666186:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628261.666192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778628261.666194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628261/real 1778628261] req@ffff954358789180 x1865013828617984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628261.666206:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628261.666213:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12596 00000100:00080000:2.0:1778628261.666218:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628261.666221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828617984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628261.666224:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628261.666240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628261.666249:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828618496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628261.666286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628261.666302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628261.666308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828617984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628261.666337:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628261.666362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628261.666373:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628261/real 1778628261] req@ffff954358788a80 x1865013828618112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628261.666406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828618112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628261.666417:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628261.666421:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628261.666425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628261.666432:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628261.666464:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628261.666476:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628261/real 1778628261] req@ffff95435878b800 x1865013828618240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628261.666491:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828618240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628261.666501:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628261.666505:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628261.666509:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628261.666525:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628261.666554:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628261.666566:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628261/real 1778628261] req@ffff95435878a300 x1865013828618368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628261.666596:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828618368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628261.666604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628261.666606:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628261.666609:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628261.666614:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628261.666628:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628261.666636:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628261/real 1778628261] req@ffff95435878ad80 x1865013828618496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628261.666645:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828618496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628261.666653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628261.666656:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628261.666658:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628261.857805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628261.857931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628262.625779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628262.881819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628262.881890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628263.649826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628263.905770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628263.905968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628264.673901:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628264.929824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628264.930000:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628265.697840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628265.953780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628265.953924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628266.721874:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628266.785864:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628266.785877:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628266.785883:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628266.785897:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12601 00000100:00080000:2.0:1778628266.785913:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628266.785922:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628266.785979:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828618624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628266.786026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628266.786031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628266.786033:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628266.786039:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12601 00000100:00080000:2.0:1778628266.786043:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628266.786046:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628266.786071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628266.786086:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828618752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628266.786105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628266.786112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628266.786118:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628266.786130:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828618624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628266.786137:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12601 00000100:00080000:2.0:1778628266.786147:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628266.786154:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628266.786165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628266.786227:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828618880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628266.786241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628266.786245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628266.786248:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778628266.786257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628266.786258:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12601 00000100:00080000:2.0:1778628266.786270:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628266.786276:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628266.786284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628266.786300:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828619008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628266.786311:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628266.786316:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628266.786318:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628266.786323:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12601 00000100:00080000:2.0:1778628266.786336:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628266.786340:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628266.786345:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628266.786365:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628266/real 1778628266] req@ffff95435878a680 x1865013828618624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628266.786366:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828619136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628266.786440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828618624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628266.786459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628266.786466:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628266.786472:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828618624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628266.786521:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628266.786562:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628266.786577:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628266/real 1778628266] req@ffff954358789f80 x1865013828618752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628266.786592:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828618752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628266.786602:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628266.786606:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628266.786610:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628266.786622:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628266.786638:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628266.786647:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628266/real 1778628266] req@ffff95435878ad80 x1865013828618880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628266.786746:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828618880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628266.786763:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628266.786788:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628266.786794:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828618880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628266.786807:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628266.786870:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628266.786884:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628266/real 1778628266] req@ffff95435878a300 x1865013828619008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628266.786897:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828619008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628266.786906:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628266.786910:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628266.786913:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628266.786920:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628266.786944:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628266.786955:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628266/real 1778628266] req@ffff95435878b800 x1865013828619136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628266.786966:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828619136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628266.786998:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628266.787004:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628266.787010:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628266.977891:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628266.978033:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628267.745881:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628268.002103:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628268.002485:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628268.769775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778628269.025997:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628269.026092:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628269.793814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628270.050218:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628270.050277:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628270.817789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778628271.074252:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628271.074391:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628271.841776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628271.905800:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628271.905810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628271.905815:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628271.905825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12606 00000100:00080000:0.0:1778628271.905838:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628271.905844:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628271.905886:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828619264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628271.905914:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628271.905917:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628271.905919:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628271.905922:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12606 00000100:00080000:0.0:1778628271.905925:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628271.905928:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628271.905949:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828619264:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778628271.905957:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628271.905970:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828619392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628271.905978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628271.905980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628271.905982:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628271.905985:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12606 00000100:00080000:0.0:1778628271.905988:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628271.905991:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628271.905995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628271.906002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828619520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628271.906008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628271.906011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628271.906012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628271.906015:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12606 00000100:00080000:0.0:1778628271.906017:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628271.906019:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628271.906022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778628271.906024:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628271.906032:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828619648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628271.906039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628271.906041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628271.906047:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628271.906050:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12606 00000100:00080000:0.0:1778628271.906054:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628271.906058:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628271.906061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628271.906069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828619776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628271.906073:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628271/real 1778628271] req@ffff954342f4bb80 x1865013828619264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628271.906096:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828619264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628271.906106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628271.906110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628271.906114:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828619264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628271.906135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628271.906151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628271.906159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628271/real 1778628271] req@ffff954342f49880 x1865013828619392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628271.906167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828619392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628271.906173:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628271.906175:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628271.906177:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628271.906181:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628271.906190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628271.906195:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628271/real 1778628271] req@ffff954342f4a300 x1865013828619520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628271.906214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828619520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628271.906219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628271.906221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628271.906223:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628271.906227:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628271.906241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628271.906247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628271/real 1778628271] req@ffff954342f4b480 x1865013828619648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628271.906254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828619648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628271.906259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628271.906261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628271.906263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628271.906266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628271.906279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628271.906286:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628271/real 1778628271] req@ffff954342f49c00 x1865013828619776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628271.906294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828619776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628271.906309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628271.906313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628271.906316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828619776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628272.097806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628272.097923:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628272.865794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628273.121813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628273.122296:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628273.889856:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628274.145778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628274.145842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628274.913780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628275.169822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628275.169898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628275.937792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628276.193803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628276.193884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628276.961848:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628277.025848:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628277.025863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628277.025870:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628277.025896:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12611 00000100:00080000:2.0:1778628277.025917:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628277.025930:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628277.026004:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828619904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628277.026050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628277.026055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628277.026058:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628277.026063:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12611 00000100:00080000:2.0:1778628277.026071:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628277.026077:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628277.026091:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628277.026132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828619904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628277.026157:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828620032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628277.026183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628277.026190:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628277.026194:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628277.026202:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12611 00000100:00080000:2.0:1778628277.026222:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628277.026230:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628277.026240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778628277.026249:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628277.026274:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828620160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628277.026292:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628277.026301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628277/real 1778628277] req@ffff954358788a80 x1865013828619904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628277.026302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628277.026310:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628277.026333:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12611 00000100:00080000:2.0:1778628277.026339:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628277.026345:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628277.026356:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628277.026358:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828619904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628277.026387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778628277.026388:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828620288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628277.026407:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628277.026427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828619904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628277.026430:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628277.026446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628277.026450:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628277.026466:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12611 00000100:00080000:2.0:1778628277.026473:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628277.026480:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620032:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628277.026488:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628277.026503:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628277.026524:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828620416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628277.026553:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628277.026605:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628277/real 1778628277] req@ffff954358789180 x1865013828620032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628277.026649:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828620032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628277.026661:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628277.026666:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628277.026704:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628277.026714:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628277.026745:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628277.026760:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628277/real 1778628277] req@ffff954358788e00 x1865013828620160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628277.026775:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828620160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628277.026785:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628277.026790:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628277.026794:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628277.026801:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628277.026820:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628277.026836:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628277/real 1778628277] req@ffff954358789c00 x1865013828620288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628277.026859:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828620288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628277.026899:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628277.026906:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628277.026913:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628277.026949:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628277.026980:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628277.026996:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628277/real 1778628277] req@ffff954358789880 x1865013828620416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628277.027010:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828620416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628277.027021:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628277.027026:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628277.027029:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628277.217781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628277.217947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628277.985791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628278.241807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628278.241882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628279.009827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628279.265853:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628279.265985:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628280.033826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628280.289822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628280.289935:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628281.057798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628281.313826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628281.313893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628282.081841:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628282.145735:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628282.145746:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628282.145752:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628282.145762:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12616 00000100:00080000:2.0:1778628282.145776:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628282.145783:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628282.145839:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828620544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628282.145874:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628282.145878:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628282.145880:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628282.145885:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12616 00000100:00080000:2.0:1778628282.145889:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628282.145894:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628282.145928:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628282.145944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828620672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628282.145957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628282.145961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628282.145963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628282.145964:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828620544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628282.145967:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12616 00000100:00080000:2.0:1778628282.145972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628282.145983:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628282.145989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628282.146000:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828620800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628282.146011:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628282.146014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628282.146017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628282.146023:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12616 00000100:00080000:2.0:1778628282.146027:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628282.146032:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628282.146039:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628282.146054:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828620928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628282.146064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628282.146067:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1778628282.146074:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628282.146076:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628282.146084:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12616 00000100:00080000:2.0:1778628282.146093:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628282.146098:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628282.146107:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628282.146122:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828621056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628282.146135:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628282/real 1778628282] req@ffff954358789500 x1865013828620544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628282.146161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828620544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628282.146189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628282.146194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628282.146199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828620544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628282.146225:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628282.146251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628282.146263:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628282/real 1778628282] req@ffff954358788700 x1865013828620672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628282.146293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828620672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628282.146304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628282.146308:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628282.146312:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628282.146320:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628282.146341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628282.146352:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628282/real 1778628282] req@ffff95435878b100 x1865013828620800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628282.146365:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828620800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628282.146375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628282.146379:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628282.146382:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628282.146399:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628282.146423:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628282.146435:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628282/real 1778628282] req@ffff95435878bb80 x1865013828620928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628282.146490:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828620928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628282.146503:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628282.146508:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628282.146513:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828620928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628282.146522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628282.146553:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628282.146568:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628282/real 1778628282] req@ffff95435878b480 x1865013828621056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628282.146580:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828621056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628282.146589:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628282.146592:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628282.146595:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628282.337848:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628282.337937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628283.105844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628283.361848:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628283.361978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628284.129851:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628284.385799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628284.385894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628285.153792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628285.409804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628285.409920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628286.177793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628286.433825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628286.433912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628287.201806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628287.265922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628287.265935:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628287.265940:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628287.265967:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12621 00000100:00080000:3.0:1778628287.265981:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628287.265991:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628287.266037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828621184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628287.266075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628287.266095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628287.266099:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628287.266106:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12621 00000100:00080000:3.0:1778628287.266113:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628287.266120:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628287.266143:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628287.266157:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828621184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628287.266166:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828621312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628287.266183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628287.266188:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628287.266190:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628287.266196:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12621 00000100:00080000:3.0:1778628287.266201:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628287.266206:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628287.266212:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628287.266225:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828621440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628287.266238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628287.266242:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628287.266245:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778628287.266255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628287.266255:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12621 00000100:00080000:3.0:1778628287.266266:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628287.266271:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628287.266287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628287.266306:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828621568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628287.266319:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628287.266325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628287.266328:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628287.266330:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628287/real 1778628287] req@ffff95434701f100 x1865013828621184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628287.266335:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12621 00000100:00080000:3.0:1778628287.266344:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628287.266354:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628287.266397:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628287.266413:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828621184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628287.266417:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828621696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628287.266473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628287.266487:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628287.266498:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828621184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628287.266541:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628287.266582:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628287.266606:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628287/real 1778628287] req@ffff95434701c380 x1865013828621312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628287.266632:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828621312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628287.266651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628287.266658:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628287.266666:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628287.266740:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628287.266772:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628287.266788:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628287/real 1778628287] req@ffff95434701d500 x1865013828621440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628287.266815:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828621440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628287.266826:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628287.266831:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628287.266835:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628287.266842:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628287.266867:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628287.266882:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628287/real 1778628287] req@ffff95434701f800 x1865013828621568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628287.266905:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828621568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628287.266921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628287.266927:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628287.266934:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628287.266946:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628287.266992:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628287.267011:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628287/real 1778628287] req@ffff95434701c700 x1865013828621696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628287.267027:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828621696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628287.267065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628287.267069:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628287.267074:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628287.457837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628287.457973:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628288.225796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628288.481900:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628288.482011:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628289.249765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628289.505763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628289.505909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628290.273810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628290.529854:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628290.529941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628291.297774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628291.553785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628291.553942:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628292.321747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628292.385887:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628292.385913:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628292.385920:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628292.385934:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12626 00000100:00080000:2.0:1778628292.385951:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628292.385971:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628292.386024:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828621824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628292.386083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628292.386119:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628292.386124:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628292.386134:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12626 00000100:00100000:0.0:1778628292.386146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828621824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628292.386151:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628292.386165:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628292.386181:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628292.386203:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828621952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628292.386221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628292.386226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628292.386228:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628292.386233:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12626 00000100:00080000:2.0:1778628292.386238:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628292.386245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628292.386247:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628292.386260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628292.386338:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828622080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628292.386346:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628292/real 1778628292] req@ffff95435878aa00 x1865013828621824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628292.386367:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628292.386379:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628292.386382:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628292.386391:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12626 00000100:00100000:0.0:1778628292.386393:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828621824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628292.386416:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628292.386435:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778628292.386444:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628292.386456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628292.386462:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828621824:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778628292.386464:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628292.386488:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828622208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628292.386501:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621952:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628292.386510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628292.386533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628292.386537:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628292.386545:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12626 00000100:00080000:2.0:1778628292.386553:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628292.386559:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628292.386568:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778628292.386570:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628292.386601:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828622336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628292.386606:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628292/real 1778628292] req@ffff954358788380 x1865013828621952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628292.386657:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828621952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628292.386727:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628292.386731:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628292.386737:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828621952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628292.386745:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628292.386773:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628292.386782:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628292/real 1778628292] req@ffff95435878b480 x1865013828622080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628292.386792:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828622080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628292.386798:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628292.386801:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628292.386804:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628292.386819:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628292.386832:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628292.386850:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628292/real 1778628292] req@ffff95435878bb80 x1865013828622208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628292.386860:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828622208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628292.386866:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628292.386886:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628292.386896:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628292.386903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628292.386921:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628292.386929:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628292/real 1778628292] req@ffff95435878b100 x1865013828622336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628292.386954:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828622336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628292.386960:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628292.386962:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628292.386964:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622336:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778628292.577801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628292.577804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628293.345794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628293.601819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628293.601967:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628294.369801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628294.625847:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628294.625874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628295.393832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778628295.649890:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628295.650035:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628296.417788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628296.673899:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628296.673940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628297.441798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628297.505828:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628297.505836:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628297.505840:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628297.505848:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12632 00000100:00080000:3.0:1778628297.505858:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628297.505864:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628297.505896:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828622464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628297.505922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628297.505925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628297.505926:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628297.505929:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12632 00000100:00080000:3.0:1778628297.505947:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628297.505951:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628297.505959:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628297.505971:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828622592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628297.505972:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828622464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628297.505986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628297.505988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628297.505989:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628297.505992:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12632 00000100:00080000:3.0:1778628297.505994:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628297.505997:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628297.506000:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628297.506009:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828622720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628297.506015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628297.506017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628297.506018:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628297.506020:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12632 00000100:00080000:3.0:1778628297.506023:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628297.506025:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628297.506028:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628297.506035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828622848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628297.506042:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628297.506064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628297.506067:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628297.506069:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628297.506072:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12632 00000100:00080000:3.0:1778628297.506076:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628297.506078:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628297.506082:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628297.506090:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828622976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628297.506095:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628297/real 1778628297] req@ffff95434701c000 x1865013828622464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628297.506118:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828622464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628297.506129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628297.506142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628297.506147:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828622464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628297.506205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628297.506227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628297.506239:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628297/real 1778628297] req@ffff95434701ea00 x1865013828622592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628297.506255:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828622592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628297.506266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628297.506270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628297.506299:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628297.506307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628297.506329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628297.506338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628297/real 1778628297] req@ffff95434701d880 x1865013828622720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628297.506347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828622720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628297.506353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628297.506356:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628297.506358:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628297.506363:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628297.506385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628297.506392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628297/real 1778628297] req@ffff95434701df80 x1865013828622848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628297.506400:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828622848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628297.506422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628297.506425:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628297.506428:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628297.506435:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628297.506454:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628297.506464:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628297/real 1778628297] req@ffff95434701e680 x1865013828622976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628297.506477:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828622976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628297.506486:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628297.506489:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628297.506493:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828622976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628297.697862:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628297.697925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628298.465796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628298.721786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628298.721810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628299.489806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628299.745834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628299.745938:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628300.513815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628300.769806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628300.769963:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628301.537813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628301.793783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628301.793917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628302.561853:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628302.625809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628302.625822:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628302.625829:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628302.625842:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12637 00000100:00080000:2.0:1778628302.625860:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628302.625893:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628302.625953:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828623104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628302.625986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628302.625991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628302.625992:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628302.625997:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12637 00000100:00080000:2.0:1778628302.626002:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628302.626007:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628302.626020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628302.626039:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828623232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628302.626055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628302.626079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628302.626082:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628302.626089:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12637 00000100:00080000:2.0:1778628302.626121:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628302.626134:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628302.626138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828623104:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778628302.626145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628302.626161:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828623360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628302.626175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628302.626194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628302.626211:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628302.626221:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12637 00000100:00080000:2.0:1778628302.626231:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628302.626239:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628302.626263:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628302.626286:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828623488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628302.626303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628302.626309:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628302.626311:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628302.626317:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12637 00000100:00080000:2.0:1778628302.626325:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628302.626332:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628302.626342:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628302.626364:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628302.626367:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828623616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628302.626476:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628302.626514:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628302.626536:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628302.817809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628302.818015:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628303.585789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628303.841791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628303.841898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628304.609817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628304.865808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628304.865989:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628305.633789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778628305.698083:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778628305.698135:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778628305.698157:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628305.698207:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628305.698217:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628305.698227:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628305.698235:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628305.698310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628302/real 1778628305] req@ffff954358789880 x1865013828623360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628305.698347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828623360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628305.698363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628305.698369:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628305.698375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628305.698384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628302/real 1778628305] req@ffff954358789c00 x1865013828623488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628305.698419:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828623488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628305.698435:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628305.698442:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628305.698448:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628305.698483:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628302/real 1778628305] req@ffff954358788e00 x1865013828623616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628305.698511:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828623616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628305.698529:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628305.698537:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628305.698543:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628305.698557:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628302/real 1778628305] req@ffff954358788700 x1865013828623104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628305.698582:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828623104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628305.698598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628305.698605:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628305.698610:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828623104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628305.698622:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628302/real 1778628305] req@ffff954358789500 x1865013828623232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628305.698655:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828623232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628305.698715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628305.698723:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628305.698729:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628305.889855:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628305.889919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628306.657799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628306.913875:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628306.913945:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628307.681782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628307.745882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628307.745894:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628307.745900:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628307.745913:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12642 00000100:00080000:3.0:1778628307.745926:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628307.745936:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628307.746026:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828623744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628307.746071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628307.746077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628307.746081:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628307.746088:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12642 00000100:00080000:3.0:1778628307.746094:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628307.746101:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628307.746144:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628307.746153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828623744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628307.746186:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828623872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628307.746208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628307.746215:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628307.746219:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628307.746226:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12642 00000100:00080000:3.0:1778628307.746235:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628307.746242:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628307.746252:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778628307.746271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628307.746279:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828624000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628307.746299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628307.746304:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628307.746307:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628307.746313:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12642 00000100:00080000:3.0:1778628307.746320:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628307.746325:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628307.746335:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628307/real 1778628307] req@ffff95434701ca80 x1865013828623744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778628307.746336:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628307.746369:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828624128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628307.746385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828623744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628307.746398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628307.746418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628307.746427:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778628307.746439:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628307.746443:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778628307.746452:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12642 00000100:00100000:0.0:1778628307.746458:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828623744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628307.746465:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628307.746477:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628307.746498:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628307.746503:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778628307.746528:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828624256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628307.746555:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628307.746584:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628307/real 1778628307] req@ffff95434701d180 x1865013828623872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628307.746609:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828623872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628307.746637:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628307.746644:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628307.746651:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828623872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628307.746663:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628307.746776:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628307.746799:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628307/real 1778628307] req@ffff95434701f480 x1865013828624000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628307.746824:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828624000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628307.746840:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628307.746847:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628307.746854:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628307.746866:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628307.746908:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628307.746929:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628307/real 1778628307] req@ffff95434701e680 x1865013828624128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628307.746952:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828624128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628307.746999:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628307.747006:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628307.747012:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628307.747045:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628307.747084:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628307.747106:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628307/real 1778628307] req@ffff95434701df80 x1865013828624256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628307.747155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828624256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628307.747171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628307.747177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628307.747183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628307.937814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628307.937890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628308.705873:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628308.961840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628308.962057:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628309.729781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628309.985831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628309.985958:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628310.753745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628311.009788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628311.009927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628311.777729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628312.033764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628312.033870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628312.801781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628312.865846:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628312.865888:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628312.865895:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628312.865923:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12647 00000100:00080000:2.0:1778628312.865934:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628312.865942:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628312.866007:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828624384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628312.866053:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628312.866061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628312.866065:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628312.866073:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12647 00000100:00080000:2.0:1778628312.866080:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628312.866088:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628312.866104:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628312.866126:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828624512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628312.866142:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828624384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628312.866144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628312.866160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628312.866163:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628312.866168:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12647 00000100:00080000:2.0:1778628312.866173:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628312.866180:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628312.866190:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628312.866236:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828624640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628312.866238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628312.866264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628312.866271:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628312.866276:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628312.866288:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12647 00000100:00100000:0.0:1778628312.866289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628312/real 1778628312] req@ffff954358789180 x1865013828624384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628312.866296:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628312.866309:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628312.866342:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628312.866351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828624384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628312.866364:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828624768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628312.866375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628312.866389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628312.866393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628312.866396:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628312.866400:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12647 00000100:00080000:2.0:1778628312.866406:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628312.866409:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628312.866416:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628312.866426:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628312.866430:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828624384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628312.866442:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828624896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628312.866497:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628312.866528:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628312.866540:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628312/real 1778628312] req@ffff954358788a80 x1865013828624512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628312.866561:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828624512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628312.866568:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628312.866571:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628312.866574:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628312.866579:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628312.866596:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628312.866604:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628312/real 1778628312] req@ffff95435878b800 x1865013828624640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628312.866614:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828624640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628312.866620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628312.866622:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628312.866624:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628312.866629:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628312.866651:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628312.866662:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628312/real 1778628312] req@ffff95435878a300 x1865013828624768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628312.866706:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828624768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628312.866733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628312.866738:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628312.866741:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628312.866747:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628312.866803:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628312.866826:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628312/real 1778628312] req@ffff95435878ad80 x1865013828624896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628312.866850:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828624896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628312.866894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628312.866901:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628312.866907:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828624896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628313.057892:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628313.057965:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628313.825763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628314.081790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628314.081905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628314.849798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628315.105785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628315.105963:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628315.873763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628316.129808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628316.129918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628316.897853:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628317.153798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628317.153957:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628317.921846:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628317.985867:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628317.985904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628317.985911:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628317.985927:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12652 00000100:00080000:2.0:1778628317.985944:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628317.985954:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628317.986014:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828625024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628317.986067:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628317.986072:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628317.986075:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628317.986081:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12652 00000100:00080000:2.0:1778628317.986086:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628317.986090:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628317.986104:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628317.986121:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828625152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628317.986138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628317.986142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628317.986148:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628317.986152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828625024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628317.986158:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12652 00000100:00080000:2.0:1778628317.986170:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628317.986175:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628317.986209:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628317.986232:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828625280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628317.986246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628317.986251:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628317.986253:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628317.986261:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12652 00000800:00000100:0.0:1778628317.986267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628317.986273:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628317.986283:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628317.986302:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628317.986320:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828625408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628317.986333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628317.986338:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778628317.986343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628317/real 1778628317] req@ffff954358789f80 x1865013828625024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628317.986343:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628317.986354:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12652 00000100:00080000:2.0:1778628317.986365:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628317.986377:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628317.986383:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628317.986398:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828625536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628317.986406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828625024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628317.986439:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628317.986447:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628317.986455:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828625024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628317.986508:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628317.986558:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628317.986584:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628317/real 1778628317] req@ffff95435878a680 x1865013828625152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628317.986610:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828625152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628317.986622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628317.986627:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628317.986631:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628317.986640:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628317.986668:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628317.986738:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628317/real 1778628317] req@ffff95435878ad80 x1865013828625280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628317.986794:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828625280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628317.986821:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628317.986827:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628317.986833:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628317.986861:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628317.986938:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628317.986974:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628317/real 1778628317] req@ffff95435878a300 x1865013828625408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628317.986999:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828625408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628317.987015:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628317.987021:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628317.987028:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628317.987040:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628317.987087:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628317.987105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628317/real 1778628317] req@ffff95435878b800 x1865013828625536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628317.987128:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828625536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628317.987145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628317.987151:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628317.987157:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628318.177850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628318.177967:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628318.945801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628319.201859:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628319.201965:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628319.969796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628320.225785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628320.225892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628320.993921:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628321.249823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628321.249999:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628322.017887:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628322.273745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628322.273797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628323.041794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628323.105862:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628323.105875:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628323.105882:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628323.105897:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12657 00000100:00080000:1.0:1778628323.105914:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628323.105925:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628323.105984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828625664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628323.106034:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628323.106039:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628323.106042:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628323.106047:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12657 00000100:00080000:1.0:1778628323.106053:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628323.106058:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628323.106070:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628323.106093:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828625792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628323.106109:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628323.106114:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628323.106118:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628323.106123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828625664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628323.106127:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12657 00000100:00080000:1.0:1778628323.106138:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628323.106144:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628323.106150:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628323.106191:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828625920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628323.106204:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628323.106208:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628323.106211:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628323.106215:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12657 00000100:00080000:1.0:1778628323.106221:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628323.106226:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628323.106232:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628323.106241:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628323.106251:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828626048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628323.106273:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628323.106277:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628323.106280:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628323.106285:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12657 00000100:00080000:1.0:1778628323.106289:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628323.106292:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628323.106299:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628323.106319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628323/real 1778628323] req@ffff954350197100 x1865013828625664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628323.106320:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828626176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628323.106366:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828625664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628323.106381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628323.106388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628323.106393:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828625664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628323.106420:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628323.106448:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628323.106471:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628323/real 1778628323] req@ffff954350194700 x1865013828625792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628323.106485:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828625792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628323.106495:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628323.106499:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628323.106503:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628323.106510:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628323.106535:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628323.106544:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628323/real 1778628323] req@ffff954350197800 x1865013828625920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628323.106573:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828625920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628323.106584:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628323.106587:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628323.106591:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828625920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628323.106599:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628323.106622:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628323.106633:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628323/real 1778628323] req@ffff954350196d80 x1865013828626048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628323.106647:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828626048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628323.106760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628323.106766:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628323.106770:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628323.106779:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628323.106811:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628323.106825:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628323/real 1778628323] req@ffff954350197480 x1865013828626176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628323.106840:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828626176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628323.106869:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628323.106873:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628323.106877:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628323.297795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628323.297929:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628324.065796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628324.321758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628324.322011:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628325.089821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628325.345855:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628325.346031:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628326.113856:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628326.369807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628326.369868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628327.137817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628327.393808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628327.393915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628328.161819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628328.225848:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628328.225859:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628328.225865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628328.225877:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12662 00000100:00080000:0.0:1778628328.225894:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628328.225904:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628328.225965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828626304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628328.226003:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628328.226007:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628328.226009:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628328.226014:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12662 00000100:00080000:0.0:1778628328.226018:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628328.226022:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628328.226034:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628328.226050:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828626432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628328.226078:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828626304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628328.226110:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628328.226114:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628328.226115:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628328.226122:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12662 00000100:00080000:0.0:1778628328.226129:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628328.226135:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628328.226145:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628328.226165:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828626560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628328.226180:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778628328.226181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628328.226190:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628328.226193:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628328.226202:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12662 00000100:00080000:0.0:1778628328.226213:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628328.226218:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628328.226227:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628328.226235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628328/real 1778628328] req@ffff954342f48700 x1865013828626304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628328.226250:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828626688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628328.226263:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628328.226267:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628328.226269:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628328.226276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828626304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628328.226278:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12662 00000100:00080000:0.0:1778628328.226297:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628328.226310:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628328.226317:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778628328.226317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628328.226334:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628328.226342:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828626304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628328.226345:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828626816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628328.226404:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628328.226446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628328.226468:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628328/real 1778628328] req@ffff954342f4b100 x1865013828626432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628328.226489:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828626432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628328.226503:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628328.226509:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628328.226532:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628328.226544:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628328.226601:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628328.226618:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628328/real 1778628328] req@ffff954342f49f80 x1865013828626560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628328.226661:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828626560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628328.226715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628328.226721:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628328.226727:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628328.226738:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628328.226773:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628328.226791:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628328/real 1778628328] req@ffff954342f49500 x1865013828626688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628328.226811:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828626688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628328.226824:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628328.226829:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628328.226834:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628328.226845:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628328.226876:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628328.226903:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628328/real 1778628328] req@ffff954342f4ad80 x1865013828626816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628328.226923:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828626816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628328.226937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628328.226943:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628328.226948:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828626816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628328.417832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628328.417925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628329.185795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628329.441827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628329.441935:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628330.209765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628330.465829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628330.465917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628331.233790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628331.489775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628331.489888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628332.257759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628332.513777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628332.513811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628333.281785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628333.345838:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628333.345862:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628333.345868:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628333.345881:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12667 00000100:00080000:2.0:1778628333.345895:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628333.345903:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628333.345956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828626944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628333.345996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628333.346002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628333.346005:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628333.346012:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12667 00000100:00080000:2.0:1778628333.346018:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628333.346052:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628333.346059:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828626944:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778628333.346071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628333.346096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828627072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628333.346109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628333.346114:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628333.346116:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628333.346120:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12667 00000100:00080000:2.0:1778628333.346126:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628333.346130:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628333.346147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628333.346164:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828627200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628333.346166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628333.346186:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628333.346191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628333.346194:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628333.346203:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12667 00000100:00080000:2.0:1778628333.346209:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628333.346216:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628333.346225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628333/real 1778628333] req@ffff954358788a80 x1865013828626944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778628333.346226:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628333.346253:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828627328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628333.346274:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628333.346277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828626944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628333.346317:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628333.346321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628333.346322:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628333.346349:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12667 00000100:00080000:0.0:1778628333.346350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628333.346362:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628333.346364:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828626944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628333.346377:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628333.346405:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628333.346424:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628333.346426:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828627456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628333.346513:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628333.346543:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628333/real 1778628333] req@ffff954358789180 x1865013828627072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628333.346584:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828627072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628333.346594:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628333.346598:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628333.346602:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628333.346610:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628333.346635:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628333.346645:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628333/real 1778628333] req@ffff954358789500 x1865013828627200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628333.346657:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828627200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628333.346665:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628333.346668:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628333.346700:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628333.346707:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628333.346728:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628333.346738:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628333/real 1778628333] req@ffff954358788700 x1865013828627328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628333.346750:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828627328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628333.346773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628333.346776:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628333.346780:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628333.346802:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628333.346823:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628333.346833:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628333/real 1778628333] req@ffff954358788e00 x1865013828627456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628333.346844:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828627456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628333.346882:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628333.346887:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628333.346891:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628333.537867:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628333.537958:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628334.305763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628334.561913:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628334.562859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628335.329743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628335.585892:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628335.585943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628336.353801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628336.610184:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628336.610414:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628337.377833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628337.633842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628337.634296:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628338.401803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628338.465882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628338.465896:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628338.465903:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628338.465918:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12672 00000100:00080000:3.0:1778628338.465937:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628338.465948:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628338.466054:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828627584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628338.466105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628338.466110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628338.466113:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628338.466119:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12672 00000100:00080000:3.0:1778628338.466124:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628338.466128:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628338.466160:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628338.466181:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828627712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628338.466195:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628338.466201:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628338.466222:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628338.466232:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12672 00000100:00100000:0.0:1778628338.466236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828627584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628338.466237:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628338.466248:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628338.466261:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628338.466333:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828627840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628338.466348:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628338.466375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628338.466379:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628338.466385:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12672 00000100:00080000:3.0:1778628338.466393:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628338.466400:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628338.466425:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628338.466436:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628338.466447:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828627968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628338.466462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628338.466467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628338.466469:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628338.466474:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12672 00000100:00080000:3.0:1778628338.466479:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628338.466498:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628338.466509:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628338/real 1778628338] req@ffff95434701d880 x1865013828627584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778628338.466510:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628338.466543:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828628096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628338.466559:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828627584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628338.466597:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628338.466607:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628338.466614:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828627584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628338.466625:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628338.466661:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628338.466679:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628338/real 1778628338] req@ffff95434701ea00 x1865013828627712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628338.466758:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828627712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628338.466770:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628338.466775:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628338.466779:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628338.466815:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628338.466844:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628338.466858:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628338/real 1778628338] req@ffff95434701c000 x1865013828627840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628338.466873:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828627840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628338.466883:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628338.466887:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628338.466891:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628338.466910:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628338.466937:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628338.466949:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628338/real 1778628338] req@ffff95434701c700 x1865013828627968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628338.466964:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828627968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628338.466975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628338.466979:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628338.466983:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828627968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628338.466990:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628338.467007:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628338.467017:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628338/real 1778628338] req@ffff95434701f800 x1865013828628096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628338.467030:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828628096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628338.467070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628338.467073:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628338.467076:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628338.657853:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628338.658005:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628339.425822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628339.681952:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628339.682638:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628340.449779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628340.705821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628340.705952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628341.473775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628341.729846:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628341.729879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628342.497745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628342.753870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628342.753872:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628343.521787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628343.585791:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628343.585840:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628343.585845:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628343.585858:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12678 00000100:00080000:3.0:1778628343.585873:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628343.585894:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628343.585938:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828628224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628343.585990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628343.585995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628343.585998:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628343.586003:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12678 00000100:00080000:3.0:1778628343.586008:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628343.586023:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628343.586035:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628343.586052:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828628352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628343.586075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628343.586086:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778628343.586097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828628224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628343.586098:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628343.586117:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12678 00000100:00080000:3.0:1778628343.586125:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628343.586134:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628343.586145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628343.586170:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828628480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628343.586205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628343.586228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628343.586235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628343.586240:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628343.586283:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12678 00000100:00100000:0.0:1778628343.586291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628343/real 1778628343] req@ffff95434701d500 x1865013828628224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628343.586297:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628343.586314:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628343.586336:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628343.586343:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828628224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628343.586363:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828628608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628343.586366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628343.586393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778628343.586394:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628343.586406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628343.586409:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628343.586413:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828628224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628343.586419:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12678 00000100:00080000:3.0:1778628343.586428:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628343.586435:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628343.586441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628343.586453:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828628736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628343.586470:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628343.586512:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628343.586537:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628343/real 1778628343] req@ffff95434701c380 x1865013828628352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628343.586553:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828628352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628343.586564:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628343.586569:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628343.586573:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628343.586581:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628343.586600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628343.586610:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628343/real 1778628343] req@ffff95434701f100 x1865013828628480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628343.586649:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828628480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628343.586659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628343.586663:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628343.586667:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628343.586738:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628343.586768:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628343.586782:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628343/real 1778628343] req@ffff95434701f800 x1865013828628608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628343.586797:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828628608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628343.586807:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628343.586811:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628343.586814:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628343.586840:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628343.586859:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628343.586869:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628343/real 1778628343] req@ffff95434701c700 x1865013828628736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628343.586883:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828628736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628343.586920:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628343.586925:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628343.586929:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628343.777895:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628343.777898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628344.545816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628344.801798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628344.801925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628345.569850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628345.825805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628345.825912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628346.593775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628346.849783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628346.849954:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628347.617803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628347.873792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628347.873968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628348.641845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628348.705798:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628348.705810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628348.705815:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628348.705825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12683 00000100:00080000:3.0:1778628348.705860:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628348.705871:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628348.705920:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828628864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628348.705964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628348.705970:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628348.705974:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628348.705981:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12683 00000100:00080000:3.0:1778628348.705987:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628348.705993:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628348.706007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628348.706050:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828628992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628348.706070:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628348.706077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628348.706081:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628348.706088:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12683 00000100:00080000:3.0:1778628348.706096:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628348.706111:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628348.706114:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828628864:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778628348.706123:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628348.706214:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828629120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628348.706236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628348.706253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628348.706258:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628348.706265:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12683 00000100:00080000:3.0:1778628348.706275:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628348.706283:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628348.706293:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628348.706309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628348.706320:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828629248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628348.706341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628348.706348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628348.706351:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628348.706359:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12683 00000100:00080000:3.0:1778628348.706379:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628348.706384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628348/real 1778628348] req@ffff95434701c000 x1865013828628864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628348.706394:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628348.706431:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628348.706461:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828628864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628348.706464:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828629376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628348.706559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628348.706576:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628348.706588:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828628864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628348.706614:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628348.706737:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628348.706769:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628348/real 1778628348] req@ffff95434701ea00 x1865013828628992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628348.706798:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828628992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628348.706838:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628348.706847:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628348.706855:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828628992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628348.706909:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628348.706962:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628348.706986:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628348/real 1778628348] req@ffff95434701d880 x1865013828629120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628348.707013:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828629120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628348.707031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628348.707038:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628348.707044:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628348.707069:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628348.707116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628348.707139:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628348/real 1778628348] req@ffff95434701df80 x1865013828629248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628348.707196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828629248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628348.707214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628348.707220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628348.707243:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628348.707258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628348.707305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628348.707329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628348/real 1778628348] req@ffff95434701e680 x1865013828629376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628348.707354:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828629376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628348.707372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628348.707379:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628348.707386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628348.897796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628348.897964:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628349.665810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628349.921826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628349.922025:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628350.689887:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628350.945806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628350.945919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628351.713773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628351.969844:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628351.969951:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628352.737773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628352.993730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628352.993824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628353.761778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628353.825833:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628353.825861:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628353.825867:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628353.825878:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12688 00000100:00080000:2.0:1778628353.825892:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628353.825901:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628353.825944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828629504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628353.825981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628353.825986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628353.825989:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628353.825997:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12688 00000100:00080000:2.0:1778628353.826004:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628353.826013:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628353.826030:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628353.826073:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828629504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628353.826088:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828629632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628353.826108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628353.826114:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628353.826118:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628353.826124:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12688 00000100:00080000:2.0:1778628353.826131:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628353.826138:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628353.826145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778628353.826152:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628353.826168:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828629760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628353.826191:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628353/real 1778628353] req@ffff954358789c00 x1865013828629504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628353.826195:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628353.826212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628353.826219:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628353.826229:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12688 00000100:00100000:0.0:1778628353.826232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828629504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628353.826240:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628353.826249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628353.826256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628353.826268:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628353.826274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828629504:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778628353.826281:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628353.826303:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828629888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628353.826309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629632:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628353.826332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628353.826336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628353.826339:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778628353.826356:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628353.826361:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12688 00000100:00100000:0.0:1778628353.826376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628353/real 1778628353] req@ffff954358789880 x1865013828629632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628353.826378:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628353.826392:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628353.826399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628353.826400:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828629632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628353.826423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628353.826432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628353.826432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828630016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628353.826437:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628353.826455:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628353.826479:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628353.826497:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628353/real 1778628353] req@ffff95435878b100 x1865013828629760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628353.826509:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828629760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628353.826516:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628353.826519:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628353.826522:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628353.826536:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628353.826552:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628353.826574:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628353/real 1778628353] req@ffff95435878bb80 x1865013828629888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628353.826586:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828629888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628353.826594:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628353.826597:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628353.826600:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828629888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628353.826606:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628353.826621:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628353.826629:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628353/real 1778628353] req@ffff95435878b480 x1865013828630016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628353.826639:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828630016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628353.826648:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628353.826650:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628353.826653:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628354.017843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628354.017873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628354.785766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628355.041780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628355.041920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628355.809821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628356.065781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628356.065930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628356.833771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628357.089784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628357.089858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628357.857738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628358.113760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628358.113873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628358.881775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628358.945780:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628358.945790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628358.945795:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628358.945807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12693 00000100:00080000:2.0:1778628358.945821:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628358.945830:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628358.945879:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828630144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628358.945927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628358.945934:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628358.945938:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628358.945976:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12693 00000100:00080000:2.0:1778628358.945984:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628358.945993:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628358.946010:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628358.946036:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828630144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628358.946044:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828630272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628358.946063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628358.946070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628358.946078:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628358.946084:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12693 00000100:00080000:2.0:1778628358.946089:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628358.946094:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628358.946102:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628358.946137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828630400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628358.946150:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628358.946157:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1778628358.946161:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628358.946162:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628358.946174:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12693 00000100:00080000:2.0:1778628358.946182:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628358.946188:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628358.946197:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628358.946214:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828630528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628358.946220:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628358/real 1778628358] req@ffff954358788380 x1865013828630144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628358.946257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628358.946262:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628358.946265:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628358.946275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828630144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628358.946276:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12693 00000100:00080000:2.0:1778628358.946287:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628358.946300:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778628358.946313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1778628358.946313:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778628358.946325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628358.946332:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828630656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628358.946339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828630144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628358.946395:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628358.946440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628358.946463:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628358/real 1778628358] req@ffff95435878aa00 x1865013828630272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628358.946488:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828630272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628358.946505:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628358.946512:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628358.946519:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628358.946531:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628358.946572:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628358.946594:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628358/real 1778628358] req@ffff95435878b480 x1865013828630400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628358.946629:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828630400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628358.946640:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628358.946644:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628358.946648:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628358.946656:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628358.946751:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628358.946775:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628358/real 1778628358] req@ffff95435878bb80 x1865013828630528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628358.946800:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828630528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628358.946817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628358.946824:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628358.946829:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628358.946838:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628358.946867:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628358.946880:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628358/real 1778628358] req@ffff95435878b100 x1865013828630656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628358.946894:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828630656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628358.946922:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628358.946928:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628358.946935:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628359.137775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628359.137824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628359.905790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628360.161801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628360.161811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628360.929796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628361.185758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628361.185865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628361.953809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628362.209738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628362.209791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628362.977753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628363.233750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628363.233763:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628364.001770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628364.065789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628364.065798:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628364.065803:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628364.065815:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12698 00000100:00080000:2.0:1778628364.065827:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628364.065836:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628364.065885:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828630784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628364.065913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628364.065916:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628364.065920:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628364.065926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12698 00000100:00080000:2.0:1778628364.065930:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628364.065935:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828630784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628364.065936:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628364.065951:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628364.065967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828630912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628364.065979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628364.065983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628364.065985:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628364.065987:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12698 00000100:00080000:2.0:1778628364.065991:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628364.065996:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628364.065996:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628364.066015:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628364.066025:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828631040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628364.066028:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628364/real 1778628364] req@ffff954358789880 x1865013828630784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628364.066042:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628364.066046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628364.066048:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628364.066052:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12698 00000100:00080000:2.0:1778628364.066056:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628364.066058:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828630784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628364.066060:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628364.066066:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628364.066074:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828631168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628364.066076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628364.066085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628364.066089:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628364.066089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628364.066092:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628364.066097:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12698 00000100:00100000:0.0:1778628364.066101:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828630784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628364.066101:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628364.066106:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628364.066112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628364.066118:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828631296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628364.066134:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628364.066166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628364.066206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628364/real 1778628364] req@ffff954358789c00 x1865013828630912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628364.066224:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828630912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628364.066236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628364.066241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628364.066245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828630912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628364.066268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628364.066290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628364.066303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628364/real 1778628364] req@ffff954358788e00 x1865013828631040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628364.066316:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828631040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628364.066324:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628364.066328:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628364.066331:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628364.066336:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628364.066351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628364.066359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628364/real 1778628364] req@ffff954358788700 x1865013828631168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628364.066366:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828631168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628364.066377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628364.066379:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628364.066382:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628364.066386:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628364.066398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628364.066404:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628364/real 1778628364] req@ffff954358789500 x1865013828631296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628364.066411:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828631296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628364.066417:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628364.066419:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628364.066420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628364.257743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628364.257775:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628365.025725:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628365.281741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628365.281767:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628366.049760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628366.305751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628366.305773:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628367.073840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628367.329776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628367.329840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628368.097835:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628368.353810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628368.353847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628369.121824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628369.185897:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628369.185919:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628369.185926:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628369.185940:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12703 00000100:00080000:3.0:1778628369.185959:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628369.185969:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628369.186028:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828631424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628369.186117:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628369.186124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628369.186128:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628369.186135:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12703 00000100:00080000:3.0:1778628369.186142:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628369.186150:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628369.186165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628369.186208:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828631552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628369.186232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628369.186239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628369.186243:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628369.186253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828631424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628369.186256:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12703 00000100:00080000:3.0:1778628369.186270:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628369.186276:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628369.186291:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628369.186317:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828631680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628369.186339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628369.186346:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628369.186350:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628369.186379:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12703 00000100:00080000:3.0:1778628369.186400:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628369.186408:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628369.186419:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628369.186443:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828631808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628369.186456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628369.186461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628369.186463:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628369.186473:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12703 00000100:00100000:0.0:1778628369.186476:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631552:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778628369.186478:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628369.186489:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628369.186509:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628369.186526:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828631936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628369.186585:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628369.186620:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628369.186646:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631936:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778628369.377807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628369.377865:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628370.145836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628370.401780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628370.401803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628371.169789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628371.425804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628371.425852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628372.193733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778628372.258058:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778628372.258105:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778628372.258120:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628372.258155:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628372.258166:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628372.258178:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628372.258189:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628372.258196:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628369/real 1778628372] req@ffff95434701ca80 x1865013828631680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628372.258220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828631680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628372.258230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628372.258234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628372.258237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628372.258243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628369/real 1778628372] req@ffff95434701e680 x1865013828631808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628372.258252:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828631808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628372.258258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628372.258274:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628372.258278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628372.258286:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628369/real 1778628372] req@ffff95434701df80 x1865013828631936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628372.258301:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828631936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628372.258311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628372.258315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628372.258319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628372.258327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628369/real 1778628372] req@ffff95434701f480 x1865013828631424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628372.258341:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828631424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628372.258349:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628372.258353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628372.258362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828631424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628372.258371:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628369/real 1778628372] req@ffff95434701d180 x1865013828631552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628372.258385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828631552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628372.258394:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628372.258423:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628372.258429:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828631552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628372.449827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628372.449904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628373.217799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628373.473808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628373.473882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628374.241772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628374.305868:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628374.305881:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628374.305886:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628374.305898:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12708 00000100:00080000:1.0:1778628374.305912:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628374.305935:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628374.305980:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828632064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628374.306030:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628374.306034:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628374.306036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628374.306040:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12708 00000100:00080000:1.0:1778628374.306045:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628374.306049:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628374.306058:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628374.306073:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828632192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628374.306084:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628374.306088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828632064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628374.306089:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628374.306092:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628374.306101:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12708 00000100:00080000:1.0:1778628374.306105:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628374.306109:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628374.306114:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628374.306127:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828632320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628374.306136:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628374.306140:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628374.306141:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628374.306145:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12708 00000100:00080000:1.0:1778628374.306156:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628374.306161:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628374.306166:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628374.306180:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828632448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628374.306181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628374.306193:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628374.306196:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628374.306198:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628374.306202:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12708 00000100:00080000:1.0:1778628374.306207:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628374.306211:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628374.306216:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628374.306228:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628374/real 1778628374] req@ffff954350194000 x1865013828632064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628374.306228:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828632576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628374.306276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828632064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628374.306292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628374.306299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628374.306304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828632064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628374.306337:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628374.306362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628374.306377:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628374/real 1778628374] req@ffff954350194380 x1865013828632192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628374.306390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828632192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628374.306400:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628374.306403:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628374.306407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628374.306425:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628374.306449:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628374.306461:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628374/real 1778628374] req@ffff954350194a80 x1865013828632320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628374.306474:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828632320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628374.306482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628374.306486:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628374.306489:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628374.306495:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628374.306518:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628374.306530:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628374/real 1778628374] req@ffff954350196a00 x1865013828632448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628374.306542:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828632448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628374.306549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628374.306552:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628374.306555:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628374.306560:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628374.306579:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628374.306590:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628374/real 1778628374] req@ffff954350194e00 x1865013828632576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628374.306612:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828632576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628374.306620:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628374.306623:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628374.306626:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628374.497748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628374.497829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628375.265778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628375.521802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628375.521948:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628376.289769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628376.545784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628376.545859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628377.313816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628377.569869:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628377.569929:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628378.337770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628378.593798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628378.593909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628379.361759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628379.425804:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628379.425815:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628379.425820:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628379.425833:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12713 00000100:00080000:1.0:1778628379.425847:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628379.425854:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628379.425901:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828632704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628379.425925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628379.425946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628379.425947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628379.425953:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12713 00000100:00080000:1.0:1778628379.425958:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628379.425962:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828632704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628379.425964:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628379.425985:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628379.426003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828632832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628379.426014:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628379.426016:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628379.426017:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628379.426021:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12713 00000100:00080000:1.0:1778628379.426042:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628379.426045:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628379.426049:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778628379.426055:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628379.426060:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828632960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628379.426075:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628379.426078:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628379.426081:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628379.426086:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12713 00000100:00080000:1.0:1778628379.426091:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628379.426096:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628379.426102:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628379/real 1778628379] req@ffff954350197b80 x1865013828632704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628379.426107:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628379.426127:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828633088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628379.426137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828632704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628379.426143:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628379.426164:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628379.426172:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628379.426185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628379.426189:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12713 00000100:00080000:0.0:1778628379.426198:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778628379.426200:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628379.426211:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628379.426212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828632704:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778628379.426220:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628379.426237:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828633216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628379.426246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628379.426268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628379.426283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628379/real 1778628379] req@ffff954350195180 x1865013828632832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628379.426312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828632832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628379.426320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628379.426323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628379.426326:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628379.426332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628379.426350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628379.426358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628379/real 1778628379] req@ffff954350195c00 x1865013828632960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628379.426369:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828632960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628379.426376:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628379.426378:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628379.426380:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828632960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628379.426385:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628379.426399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628379.426406:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628379/real 1778628379] req@ffff954350195500 x1865013828633088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628379.426415:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828633088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628379.426434:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628379.426437:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628379.426440:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628379.426456:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628379.426473:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628379.426482:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628379/real 1778628379] req@ffff954350195f80 x1865013828633216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628379.426491:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828633216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628379.426499:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628379.426502:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628379.426505:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628379.617760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628379.617980:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628380.385740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628380.641832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628380.641978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628381.409801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628381.665800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628381.665903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628382.433741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628382.689779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628382.689839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628383.457775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628383.713761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628383.713835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628384.481730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628384.545787:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628384.545797:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628384.545802:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628384.545813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12719 00000100:00080000:1.0:1778628384.545825:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628384.545832:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628384.545880:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828633344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628384.545903:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628384.545906:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628384.545907:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628384.545911:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12719 00000100:00100000:0.0:1778628384.545933:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828633344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628384.545935:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628384.545942:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628384.545952:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628384.545966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828633472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628384.545974:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628384.545977:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628384.545979:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628384.545982:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12719 00000100:00080000:1.0:1778628384.545999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628384.546002:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628384.546005:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778628384.546012:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628384.546015:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828633600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628384.546023:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628384.546026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628384.546027:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628384.546030:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12719 00000100:00080000:1.0:1778628384.546036:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628384.546038:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628384.546042:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628384.546051:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828633728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628384.546057:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628384/real 1778628384] req@ffff954350195880 x1865013828633344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628384.546082:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628384.546085:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628384.546087:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628384.546089:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828633344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628384.546092:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12719 00000100:00080000:1.0:1778628384.546095:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628384.546097:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628384.546104:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778628384.546115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778628384.546117:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828633856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628384.546131:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628384.546135:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828633344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628384.546165:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628384.546184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628384.546193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628384/real 1778628384] req@ffff954350195f80 x1865013828633472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628384.546202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828633472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628384.546208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628384.546210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628384.546212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628384.546215:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628384.546226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628384.546231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628384/real 1778628384] req@ffff954350195500 x1865013828633600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628384.546238:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828633600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628384.546243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628384.546245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628384.546247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628384.546256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628384.546271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628384.546277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628384/real 1778628384] req@ffff954350195c00 x1865013828633728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628384.546284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828633728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628384.546290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628384.546292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628384.546293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628384.546297:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628384.546307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628384.546312:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628384/real 1778628384] req@ffff954350195180 x1865013828633856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628384.546319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828633856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628384.546325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628384.546326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628384.546328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828633856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628384.737762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628384.737851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628385.505797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628385.761782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628385.761893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628386.529857:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628386.785795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628386.785854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628387.553818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628387.809819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628387.809925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628388.577790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628388.833867:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628388.833896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628389.601841:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628389.665875:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628389.665885:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628389.665890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628389.665900:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12724 00000100:00080000:3.0:1778628389.665913:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628389.665930:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628389.665968:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828633984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628389.665998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628389.666004:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628389.666007:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628389.666013:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12724 00000100:00080000:3.0:1778628389.666018:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628389.666024:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628389.666036:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628389.666055:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828634112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628389.666069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628389.666074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628389.666124:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628389.666133:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12724 00000100:00080000:3.0:1778628389.666140:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628389.666155:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628389.666164:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628389.666184:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828634240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628389.666207:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628389.666209:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828633984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628389.666217:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628389.666224:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628389.666244:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12724 00000100:00080000:3.0:1778628389.666248:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628389.666252:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628389.666258:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628389.666276:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828634368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628389.666291:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628389.666324:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628389.666326:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628389.666329:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12724 00000100:00080000:3.0:1778628389.666334:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628389.666357:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628389.666362:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628389.666383:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828634496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628389.666398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628389.666458:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628389/real 1778628389] req@ffff95434701d880 x1865013828633984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628389.666483:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828633984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628389.666496:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628389.666501:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628389.666506:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828633984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628389.666516:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628389.666541:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628389.666553:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628389/real 1778628389] req@ffff95434701ea00 x1865013828634112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628389.666565:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828634112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628389.666574:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628389.666577:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628389.666580:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628389.666610:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628389.666626:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628389.666635:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628389/real 1778628389] req@ffff95434701c000 x1865013828634240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628389.666729:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828634240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628389.666740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628389.666744:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628389.666747:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628389.666755:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628389.666782:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628389.666794:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628389/real 1778628389] req@ffff95434701c700 x1865013828634368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628389.666806:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828634368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628389.666815:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628389.666818:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628389.666821:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628389.666827:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628389.666841:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628389.666849:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628389/real 1778628389] req@ffff95434701f800 x1865013828634496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628389.666860:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828634496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628389.666887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628389.666891:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628389.666895:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628389.857845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628389.857933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628390.625755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628390.881819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628390.881877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628391.649805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628391.905840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628391.905983:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628392.673774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628392.929855:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628392.929969:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628393.697756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628393.953763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628393.953848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628394.721788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628394.785861:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628394.785870:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628394.785874:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628394.785882:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12729 00000100:00080000:2.0:1778628394.785908:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628394.785914:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628394.785942:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828634624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628394.785964:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628394.785966:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628394.785968:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628394.785971:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12729 00000100:00080000:2.0:1778628394.785973:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628394.785975:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628394.785982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628394.785990:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828634752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628394.785997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628394.786000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628394.786001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628394.786003:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12729 00000100:00080000:2.0:1778628394.786005:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628394.786009:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628394.786013:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628394.786030:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828634880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628394.786037:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628394.786043:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628394.786044:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628394.786048:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12729 00000100:00080000:2.0:1778628394.786052:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628394.786071:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628394.786076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628394.786084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828634624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628394.786094:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828635008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628394.786108:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628394.786111:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628394.786112:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628394.786117:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12729 00000100:00080000:2.0:1778628394.786120:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628394.786134:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628394.786137:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628394.786148:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828635136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628394.786208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628394.786298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628394/real 1778628394] req@ffff954358789180 x1865013828634624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628394.786317:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828634624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628394.786328:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628394.786332:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628394.786335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828634624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628394.786343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628394.786371:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628394.786381:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628394/real 1778628394] req@ffff954358788a80 x1865013828634752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628394.786391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828634752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628394.786422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628394.786425:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628394.786428:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628394.786433:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628394.786450:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628394.786457:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628394/real 1778628394] req@ffff95435878b800 x1865013828634880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628394.786466:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828634880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628394.786471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628394.786473:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628394.786475:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828634880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628394.786516:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628394.786530:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628394.786536:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628394/real 1778628394] req@ffff95435878a300 x1865013828635008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628394.786545:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828635008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628394.786551:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628394.786554:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628394.786561:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628394.786567:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628394.786582:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628394.786598:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628394/real 1778628394] req@ffff95435878ad80 x1865013828635136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628394.786607:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828635136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628394.786614:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628394.786616:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628394.786618:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628394.977783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628394.977862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628395.745745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628396.001745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628396.001797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628396.769782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628397.025809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628397.025922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628397.793739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628398.049736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628398.049824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628398.817791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628399.073775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628399.073825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628399.841766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628399.905793:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628399.905804:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628399.905828:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628399.905838:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12734 00000100:00080000:2.0:1778628399.905849:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628399.905857:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628399.905895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828635264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628399.905942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628399.905946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628399.905948:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628399.905952:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12734 00000100:00080000:2.0:1778628399.905956:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628399.905959:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628399.905969:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628399.906010:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828635392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628399.906023:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828635264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628399.906027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628399.906046:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628399.906048:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628399.906053:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12734 00000100:00080000:2.0:1778628399.906057:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628399.906061:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628399.906067:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628399.906084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828635520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628399.906157:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778628399.906166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628399.906168:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628399.906174:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628399.906183:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12734 00000100:00080000:2.0:1778628399.906192:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628399.906199:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628399.906208:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628399.906223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628399/real 1778628399] req@ffff95435878a680 x1865013828635264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628399.906228:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828635648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628399.906261:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628399.906268:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628399.906271:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628399.906278:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12734 00000100:00080000:2.0:1778628399.906284:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628399.906286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828635264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628399.906297:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778628399.906308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1778628399.906310:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778628399.906318:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628399.906331:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828635776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628399.906338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828635264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628399.906378:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628399.906418:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628399.906454:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628399/real 1778628399] req@ffff954358789f80 x1865013828635392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628399.906477:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828635392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628399.906491:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628399.906497:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628399.906503:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628399.906513:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628399.906548:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628399.906586:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628399/real 1778628399] req@ffff95435878ad80 x1865013828635520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628399.906606:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828635520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628399.906631:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628399.906636:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628399.906641:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628399.906653:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628399.906682:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628399.906763:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628399/real 1778628399] req@ffff95435878a300 x1865013828635648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628399.906778:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828635648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628399.906786:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628399.906789:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628399.906792:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628399.906799:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628399.906819:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628399.906828:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628399/real 1778628399] req@ffff95435878b800 x1865013828635776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628399.906839:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828635776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628399.906847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628399.906850:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628399.906852:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828635776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628400.097745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628400.097833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628400.865808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628401.122536:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628401.122555:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628401.889727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628402.145864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778628402.145920:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628402.913781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628403.169889:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628403.169930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628403.937796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628404.193995:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628404.194367:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628404.961749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628405.025797:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628405.025825:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628405.025830:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628405.025857:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12739 00000100:00080000:0.0:1778628405.025867:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628405.025874:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628405.025907:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828635904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628405.025953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628405.025957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628405.025959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628405.025963:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12739 00000100:00080000:0.0:1778628405.025967:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628405.025971:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628405.025981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628405.025999:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828636032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628405.026000:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828635904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628405.026010:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628405.026021:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628405.026023:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628405.026027:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12739 00000100:00080000:0.0:1778628405.026030:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628405.026053:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628405.026059:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628405.026074:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828636160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628405.026117:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628405.026122:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1778628405.026125:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628405.026127:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628405.026136:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12739 00000100:00080000:0.0:1778628405.026144:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628405.026151:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628405.026159:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628405.026168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628405/real 1778628405] req@ffff954342f4a680 x1865013828635904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628405.026180:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828636288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628405.026191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828635904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628405.026192:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628405.026203:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628405.026207:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628405.026217:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12739 00000100:00080000:2.0:1778628405.026219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628405.026221:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628405.026226:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778628405.026228:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628405.026235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828635904:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778628405.026236:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628405.026254:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828636416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628405.026274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628405.026308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628405.026325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628405/real 1778628405] req@ffff954342f48a80 x1865013828636032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628405.026348:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828636032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628405.026356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628405.026359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628405.026362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628405.026368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628405.026386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628405.026395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628405/real 1778628405] req@ffff954342f4ad80 x1865013828636160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628405.026419:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828636160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628405.026428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628405.026432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628405.026436:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628405.026444:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628405.026472:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628405.026487:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628405/real 1778628405] req@ffff954342f49500 x1865013828636288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628405.026504:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828636288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628405.026515:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628405.026518:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628405.026521:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628405.026526:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628405.026545:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628405.026554:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628405/real 1778628405] req@ffff954342f49f80 x1865013828636416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628405.026564:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828636416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628405.026583:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628405.026586:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628405.026588:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628405.217786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628405.217952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628405.985792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628406.241970:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628406.243132:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628407.009767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628407.265791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628407.265878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628408.033832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628408.289866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778628408.289878:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628409.057748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628409.313851:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628409.313930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628410.081789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628410.145899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628410.145908:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628410.145914:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628410.145924:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12744 00000100:00080000:2.0:1778628410.145958:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628410.145967:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628410.146006:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828636544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628410.146040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628410.146044:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628410.146046:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628410.146051:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12744 00000100:00080000:2.0:1778628410.146054:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628410.146057:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628410.146067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628410.146078:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828636672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628410.146103:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628410.146106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628410.146108:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628410.146116:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12744 00000100:00100000:0.0:1778628410.146118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828636544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628410.146120:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628410.146136:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628410.146142:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628410.146163:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828636800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628410.146178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628410.146183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628410.146185:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628410.146191:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12744 00000100:00080000:2.0:1778628410.146198:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628410.146207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628410.146208:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628410.146220:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628410.146239:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828636928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628410.146265:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628410.146269:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628410.146273:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628410.146274:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628410/real 1778628410] req@ffff954358788a80 x1865013828636544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628410.146280:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12744 00000100:00080000:2.0:1778628410.146296:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628410.146304:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628410.146310:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628410.146321:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828636544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628410.146327:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828637056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628410.146375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628410.146383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628410.146390:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828636544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628410.146416:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628410.146443:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628410.146456:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628410/real 1778628410] req@ffff954358789180 x1865013828636672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628410.146468:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828636672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628410.146477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628410.146480:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628410.146483:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628410.146489:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628410.146503:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628410.146511:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628410/real 1778628410] req@ffff954358789500 x1865013828636800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628410.146551:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828636800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628410.146559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628410.146562:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628410.146565:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628410.146571:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628410.146592:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628410.146601:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628410/real 1778628410] req@ffff954358788700 x1865013828636928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628410.146612:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828636928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628410.146619:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628410.146622:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628410.146625:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828636928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628410.146630:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628410.146643:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628410.146650:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628410/real 1778628410] req@ffff954358788e00 x1865013828637056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628410.146660:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828637056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628410.146708:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628410.146711:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628410.146714:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628410.337840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628410.337920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628411.105810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628411.361876:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628411.361988:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628412.129749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628412.385878:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628412.385918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628413.153786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628413.409850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628413.409952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628414.177825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628414.433903:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628414.433957:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628415.201902:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628415.265800:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628415.265813:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628415.265819:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628415.265860:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12749 00000100:00080000:2.0:1778628415.265875:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628415.265885:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628415.265941:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828637184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628415.265986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628415.265991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628415.265994:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628415.265999:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12749 00000100:00080000:2.0:1778628415.266003:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628415.266018:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628415.266030:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628415.266062:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828637312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628415.266077:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628415.266082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628415.266085:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628415.266090:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12749 00000100:00100000:0.0:1778628415.266099:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828637184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628415.266100:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628415.266113:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628415.266120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628415.266137:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828637440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628415.266151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628415.266156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628415.266158:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628415.266163:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12749 00000100:00080000:2.0:1778628415.266167:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628415.266172:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628415.266179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628415.266210:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828637568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628415.266211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628415.266228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628415.266232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628415.266236:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628415.266249:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12749 00000100:00080000:2.0:1778628415.266256:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628415.266262:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628415.266270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628415.266272:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628415/real 1778628415] req@ffff954358789c00 x1865013828637184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628415.266290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828637696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628415.266309:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828637184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628415.266326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628415.266334:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628415.266339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828637184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628415.266350:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628415.266376:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628415.266390:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628415/real 1778628415] req@ffff954358789880 x1865013828637312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628415.266405:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828637312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628415.266428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628415.266433:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628415.266438:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628415.266469:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628415.266508:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628415.266529:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628415/real 1778628415] req@ffff95435878b100 x1865013828637440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628415.266554:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828637440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628415.266571:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628415.266578:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628415.266584:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628415.266596:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628415.266631:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628415.266646:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628415/real 1778628415] req@ffff95435878bb80 x1865013828637568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628415.266661:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828637568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628415.266717:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628415.266722:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628415.266736:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628415.266745:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628415.266775:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628415.266807:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628415/real 1778628415] req@ffff95435878b480 x1865013828637696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628415.266822:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828637696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628415.266859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628415.266864:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628415.266868:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628415.457919:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628415.458046:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628416.225788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628416.481782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628416.481897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628417.249869:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628417.505842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628417.506049:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628418.273821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628418.529825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628418.530027:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628419.297798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628419.553863:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628419.554001:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628420.321805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628420.385888:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628420.385910:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628420.385920:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628420.385941:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12754 00000100:00080000:2.0:1778628420.385965:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628420.385975:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628420.386037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828637824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628420.386085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628420.386091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628420.386093:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628420.386100:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12754 00000100:00080000:2.0:1778628420.386105:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628420.386110:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628420.386146:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628420.386182:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828637952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628420.386204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828637824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628420.386209:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628420.386232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628420.386235:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628420.386242:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12754 00000100:00080000:2.0:1778628420.386249:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628420.386259:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628420.386267:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628420.386290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828638080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628420.386303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628420.386310:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628420.386312:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628420.386317:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12754 00000100:00080000:2.0:1778628420.386321:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628420.386332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628420.386339:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628420.386349:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628420.386371:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828638208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628420.386387:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628420.386395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628420/real 1778628420] req@ffff95435878aa00 x1865013828637824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628420.386397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628420.386417:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628420.386429:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12754 00000100:00100000:0.0:1778628420.386438:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828637824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628420.386438:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628420.386462:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628420.386472:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628420.386490:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828638336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628420.386491:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628420.386512:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628420.386537:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828637824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628420.386578:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628420.386612:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628420.386629:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628420/real 1778628420] req@ffff954358788380 x1865013828637952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628420.386645:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828637952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628420.386656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628420.386661:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628420.386665:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828637952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628420.386773:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628420.386805:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628420.386820:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628420/real 1778628420] req@ffff95435878b480 x1865013828638080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628420.386835:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828638080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628420.386846:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628420.386850:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628420.386853:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628420.386861:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628420.386880:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628420.386909:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628420/real 1778628420] req@ffff95435878bb80 x1865013828638208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628420.386924:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828638208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628420.386935:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628420.386939:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628420.386942:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628420.386950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628420.386971:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628420.386981:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628420/real 1778628420] req@ffff95435878b100 x1865013828638336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628420.386995:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828638336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628420.387005:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628420.387008:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628420.387012:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628420.577816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628420.578012:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628421.345809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628421.601824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628421.601916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628422.369797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628422.625837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628422.625978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628423.393767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628423.649749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628423.649827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628424.417717:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628424.673732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628424.673798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628425.441776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628425.505764:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628425.505772:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628425.505777:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628425.505787:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12760 00000100:00080000:2.0:1778628425.505799:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628425.505807:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628425.505848:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828638464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628425.505889:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628425.505894:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628425.505898:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628425.505904:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12760 00000100:00080000:2.0:1778628425.505928:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628425.505958:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628425.505976:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628425.505986:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828638464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628425.506011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828638592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628425.506038:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628425.506046:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628425.506049:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628425.506056:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12760 00000100:00080000:2.0:1778628425.506067:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628425.506075:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628425.506078:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628425.506086:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628425.506111:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828638720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628425.506125:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628425.506130:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628425.506133:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628425.506139:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12760 00000100:00080000:2.0:1778628425.506144:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628425.506151:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628425/real 1778628425] req@ffff954358789880 x1865013828638464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628425.506156:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628425.506167:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628425.506188:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828638848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628425.506200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828638464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628425.506227:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628425.506234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628425.506241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628425.506247:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778628425.506249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828638464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628425.506252:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628425.506263:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12760 00000100:00100000:0.0:1778628425.506278:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638592:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628425.506285:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628425.506295:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628425.506304:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778628425.506311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628425.506327:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828638976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628425.506329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628425/real 1778628425] req@ffff954358789c00 x1865013828638592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628425.506370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828638592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628425.506393:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628425.506397:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628425.506401:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628425.506407:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628425.506426:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628425.506436:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628425/real 1778628425] req@ffff954358788e00 x1865013828638720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628425.506447:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828638720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628425.506489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628425.506493:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628425.506497:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628425.506506:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628425.506531:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628425.506543:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628425/real 1778628425] req@ffff954358788700 x1865013828638848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628425.506557:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828638848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628425.506569:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628425.506575:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628425.506581:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628425.506615:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628425.506723:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628425.506741:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628425/real 1778628425] req@ffff954358789500 x1865013828638976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628425.506753:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828638976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628425.506762:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628425.506765:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628425.506776:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828638976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628425.697849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628425.697906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628426.465778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628426.721834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628426.721970:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628427.489766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628427.745827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628427.745968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628428.513744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628428.769849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628428.769993:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628429.537789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628429.793841:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628429.793975:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628430.561781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628430.625758:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628430.625766:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628430.625769:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628430.625777:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12765 00000100:00080000:0.0:1778628430.625787:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628430.625794:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628430.625824:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828639104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628430.625852:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628430.625855:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628430.625856:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628430.625859:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12765 00000100:00080000:0.0:1778628430.625862:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628430.625864:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628430.625871:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628430.625881:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828639232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628430.625889:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778628430.625890:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828639104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628430.625908:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628430.625910:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628430.625914:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12765 00000100:00080000:0.0:1778628430.625917:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628430.625920:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628430.625924:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628430.625933:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828639360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628430.625940:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628430.625942:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628430.625943:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628430.625947:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12765 00000800:00000100:2.0:1778628430.625952:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628430.625953:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628430.625957:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628430.625960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628430.625969:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828639488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628430.625976:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628430.625979:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628430.625980:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628430.625985:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12765 00000100:00100000:2.0:1778628430.625985:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628430/real 1778628430] req@ffff954342f4b100 x1865013828639104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628430.625987:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628430.625993:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628430.626002:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628430.626007:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828639104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628430.626012:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828639616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628430.626027:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628430.626032:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628430.626036:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828639104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628430.626062:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628430.626086:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628430.626109:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628430/real 1778628430] req@ffff954342f48700 x1865013828639232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628430.626121:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828639232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628430.626127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628430.626129:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628430.626131:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628430.626136:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628430.626151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628430.626157:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628430/real 1778628430] req@ffff954342f49c00 x1865013828639360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628430.626165:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828639360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628430.626170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628430.626172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628430.626174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628430.626178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628430.626188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628430.626192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628430/real 1778628430] req@ffff954342f4b480 x1865013828639488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628430.626199:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828639488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628430.626210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628430.626212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628430.626214:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628430.626218:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628430.626232:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628430.626238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628430/real 1778628430] req@ffff954342f4a300 x1865013828639616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628430.626245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828639616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628430.626250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628430.626252:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628430.626254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628430.817831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628430.817913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628431.585741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628431.841827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628431.841898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628432.609789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628432.865789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628432.865884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628433.633774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628433.889825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628433.889932:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628434.657797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628434.913836:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628434.913955:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628435.681796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628435.745878:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628435.745887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628435.745891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628435.745900:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12770 00000100:00080000:2.0:1778628435.745911:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628435.745918:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628435.745962:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828639744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628435.745996:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628435.745999:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628435.746001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628435.746004:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12770 00000100:00080000:2.0:1778628435.746008:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628435.746011:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628435.746031:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628435.746040:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828639872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628435.746052:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628435.746056:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628435.746058:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628435.746063:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12770 00000100:00080000:2.0:1778628435.746067:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628435.746073:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628435.746079:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628435.746097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828639744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628435.746097:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828640000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628435.746119:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628435.746123:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628435.746125:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628435.746138:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12770 00000100:00080000:2.0:1778628435.746144:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628435.746147:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628435.746152:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628435.746166:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828640128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628435.746174:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628435.746177:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628435.746178:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628435.746181:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12770 00000100:00080000:2.0:1778628435.746184:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628435.746187:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628435.746192:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628435.746201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628435.746203:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828640256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628435.746270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628435.746296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628435.746313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628435.937752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628435.937864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628436.705766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628436.961751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628436.961804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628437.729791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628437.985803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628437.985906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628438.753787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778628438.818074:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778628438.818141:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778628438.818163:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628438.818216:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628438.818231:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628438.818245:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628438.818257:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628438.818333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628435/real 1778628438] req@ffff95435878b800 x1865013828640000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628438.818403:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828640000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628438.818424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628438.818433:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628438.818440:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628438.818486:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628435/real 1778628438] req@ffff95435878a300 x1865013828640128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628438.818511:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828640128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628438.818526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628438.818532:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628438.818538:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628438.818549:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628435/real 1778628438] req@ffff95435878ad80 x1865013828640256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628438.818567:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828640256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628438.818581:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628438.818587:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628438.818592:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628438.818602:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628435/real 1778628438] req@ffff954358789180 x1865013828639744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628438.818623:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828639744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628438.818638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628438.818644:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628438.818650:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828639744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628438.818722:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628435/real 1778628438] req@ffff954358788a80 x1865013828639872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628438.818752:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828639872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628438.818767:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628438.818774:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628438.818780:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828639872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628439.009898:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628439.009922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628439.777786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628440.033817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628440.033947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628440.801765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628440.865751:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628440.865761:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628440.865765:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628440.865775:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12775 00000100:00080000:2.0:1778628440.865786:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628440.865808:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628440.865839:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828640384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628440.865866:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628440.865869:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628440.865871:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628440.865874:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12775 00000100:00080000:2.0:1778628440.865878:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628440.865881:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628440.865889:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628440.865900:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828640512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628440.865909:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628440.865912:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628440.865914:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628440.865920:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12775 00000100:00100000:0.0:1778628440.865921:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828640384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628440.865922:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628440.865929:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628440.865936:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628440.865945:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828640640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628440.865951:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628440.865954:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628440.865961:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628440.865965:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12775 00000100:00080000:2.0:1778628440.865968:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628440.865972:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628440.865978:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628440.865992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828640768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628440.866003:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628440.866007:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628440.866011:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778628440.866013:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628440.866019:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12775 00000100:00080000:2.0:1778628440.866024:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628440.866030:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628440.866036:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628440.866051:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828640896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628440.866060:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628440/real 1778628440] req@ffff954358789f80 x1865013828640384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628440.866091:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828640384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628440.866104:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628440.866110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628440.866115:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828640384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628440.866142:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628440.866167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628440.866183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628440/real 1778628440] req@ffff95435878a680 x1865013828640512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628440.866214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828640512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628440.866224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628440.866228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628440.866232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628440.866240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628440.866264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628440.866276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628440/real 1778628440] req@ffff954358788a80 x1865013828640640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628440.866290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828640640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628440.866300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628440.866303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628440.866307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628440.866315:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628440.866358:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628440.866371:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628440/real 1778628440] req@ffff954358789180 x1865013828640768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628440.866386:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828640768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628440.866410:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628440.866414:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628440.866419:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628440.866427:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628440.866453:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628440.866466:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628440/real 1778628440] req@ffff95435878ad80 x1865013828640896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628440.866479:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828640896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628440.866488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628440.866492:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628440.866495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828640896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628441.057842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628441.057919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628441.825750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628442.081830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628442.081903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628442.849845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628443.105824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628443.105943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628443.873761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628444.129763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628444.129823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628444.897792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628445.153737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628445.153787:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628445.921838:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628445.985785:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628445.985795:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628445.985800:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628445.985810:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12780 00000100:00080000:2.0:1778628445.985841:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628445.985850:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628445.985889:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828641024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628445.985927:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628445.985934:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628445.985937:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628445.985944:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12780 00000100:00080000:2.0:1778628445.985950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628445.985957:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628445.985969:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628445.985989:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828641152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628445.986001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628445.986005:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628445.986007:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628445.986011:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12780 00000100:00080000:2.0:1778628445.986016:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628445.986019:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628445.986025:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628445.986038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828641280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628445.986048:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628445.986060:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628445.986063:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628445.986068:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12780 00000100:00080000:2.0:1778628445.986072:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628445.986077:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628445.986084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628445.986097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828641024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628445.986102:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828641408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628445.986133:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628445.986136:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628445.986138:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628445.986145:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12780 00000100:00080000:2.0:1778628445.986148:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628445.986151:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628445.986155:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628445.986167:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828641536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628445.986254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628445.986315:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628445/real 1778628445] req@ffff95435878a300 x1865013828641024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628445.986345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828641024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628445.986381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628445.986387:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628445.986393:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828641024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628445.986404:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628445.986439:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628445.986454:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628445/real 1778628445] req@ffff95435878b800 x1865013828641152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628445.986470:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828641152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628445.986481:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628445.986486:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628445.986490:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628445.986496:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628445.986514:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628445.986524:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628445/real 1778628445] req@ffff954358789500 x1865013828641280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628445.986538:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828641280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628445.986548:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628445.986552:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628445.986555:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628445.986605:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628445.986631:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628445.986644:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628445/real 1778628445] req@ffff954358788700 x1865013828641408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628445.986658:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828641408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628445.986669:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628445.986722:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628445.986726:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628445.986734:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628445.986758:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628445.986771:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628445/real 1778628445] req@ffff954358788e00 x1865013828641536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628445.986785:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828641536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628445.986796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628445.986801:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628445.986804:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628446.177841:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628446.177878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628446.945795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628447.201750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628447.201918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628447.969743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628448.225750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628448.225815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628448.993793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628449.249786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628449.249912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628450.017807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628450.273800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628450.273939:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628451.041791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628451.105876:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628451.105887:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628451.105893:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628451.105904:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12785 00000100:00080000:0.0:1778628451.105919:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628451.105930:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628451.105992:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828641664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628451.106048:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628451.106054:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628451.106057:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628451.106064:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12785 00000100:00080000:0.0:1778628451.106071:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628451.106077:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628451.106090:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628451.106115:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828641792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628451.106119:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828641664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628451.106167:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628451.106172:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628451.106175:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628451.106181:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12785 00000100:00080000:0.0:1778628451.106187:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628451.106192:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628451.106226:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628451.106255:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828641920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628451.106257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628451.106282:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628451.106289:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628451.106294:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628451.106305:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12785 00000100:00080000:0.0:1778628451.106315:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778628451.106331:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628451/real 1778628451] req@ffff954342f49880 x1865013828641664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628451.106345:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628451.106362:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628451.106373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828641664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628451.106386:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828642048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628451.106399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628451.106407:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628451.106413:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778628451.106418:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628451.106425:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628451.106426:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828641664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628451.106439:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12785 00000100:00080000:0.0:1778628451.106446:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628451.106452:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628451.106457:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628451.106470:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628451.106476:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828642176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778628451.106514:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628451.106534:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628451/real 1778628451] req@ffff954342f4bb80 x1865013828641792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628451.106555:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828641792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628451.106567:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628451.106571:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628451.106575:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628451.106582:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628451.106600:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628451.106610:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628451/real 1778628451] req@ffff954342f48e00 x1865013828641920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628451.106624:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828641920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628451.106651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628451.106656:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628451.106660:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828641920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628451.106667:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628451.106730:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628451.106745:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628451/real 1778628451] req@ffff954342f4aa00 x1865013828642048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628451.106759:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828642048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628451.106769:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628451.106773:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628451.106777:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628451.106799:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628451.106818:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628451.106829:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628451/real 1778628451] req@ffff954342f49180 x1865013828642176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628451.106842:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828642176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628451.106852:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628451.106856:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628451.106872:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628451.297780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628451.297849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628452.065758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628452.321820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628452.321859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628453.089925:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628453.345821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628453.345909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628454.113784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628454.369832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628454.369905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628455.137795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628455.393770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628455.393951:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628456.161832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628456.225772:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628456.225782:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628456.225787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628456.225799:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12790 00000100:00080000:0.0:1778628456.225813:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628456.225823:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628456.225879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828642304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628456.225917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628456.225922:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628456.225925:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628456.225930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12790 00000100:00080000:0.0:1778628456.225934:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628456.225948:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628456.225960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628456.225976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828642432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628456.225991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628456.226000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778628456.226001:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828642304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628456.226003:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628456.226018:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12790 00000100:00080000:0.0:1778628456.226054:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628456.226089:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628456.226098:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628456.226120:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828642560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628456.226134:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628456.226143:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1778628456.226145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628456.226146:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628456.226158:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12790 00000100:00080000:0.0:1778628456.226168:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628456.226173:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628456.226181:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628456.226198:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828642688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628456.226206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628456/real 1778628456] req@ffff954342f48000 x1865013828642304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628456.226222:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628456.226233:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628456.226239:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628456.226248:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12790 00000100:00080000:0.0:1778628456.226258:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628456.226261:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628456.226267:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628456.226289:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828642304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628456.226295:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828642816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628456.226361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628456.226371:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628456.226377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828642304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628456.226412:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628456.226448:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628456.226465:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628456/real 1778628456] req@ffff954342f48380 x1865013828642432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628456.226482:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828642432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628456.226493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628456.226497:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628456.226513:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628456.226522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628456.226548:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628456.226560:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628456/real 1778628456] req@ffff954342f49180 x1865013828642560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628456.226575:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828642560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628456.226585:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628456.226589:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628456.226593:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628456.226600:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628456.226618:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628456.226628:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628456/real 1778628456] req@ffff954342f4aa00 x1865013828642688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628456.226641:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828642688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628456.226651:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628456.226655:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628456.226658:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628456.226665:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628456.226743:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628456.226789:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628456/real 1778628456] req@ffff954342f48e00 x1865013828642816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628456.226806:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828642816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628456.226817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628456.226821:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628456.226835:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828642816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628456.417781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628456.417899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628457.185760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628457.441753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628457.441816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628458.209735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628458.465752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628458.465838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628459.233741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628459.489759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628459.489828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628460.257755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628460.513795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628460.513912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628461.281837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628461.345845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628461.345853:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628461.345858:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628461.345873:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12795 00000100:00080000:1.0:1778628461.345882:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628461.345889:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628461.345937:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828642944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628461.345968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628461.345971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628461.345973:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628461.345976:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12795 00000100:00080000:1.0:1778628461.345979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628461.345982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628461.345990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628461.346001:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828643072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628461.346009:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828642944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628461.346012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628461.346029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628461.346031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628461.346035:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12795 00000100:00080000:1.0:1778628461.346038:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628461.346041:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628461.346046:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628461.346055:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828643200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628461.346062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628461.346064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628461.346065:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778628461.346070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628461.346073:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12795 00000100:00080000:1.0:1778628461.346077:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628461.346079:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628461.346083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628461.346092:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828643328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628461.346098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628461.346101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628461.346102:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628461.346106:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12795 00000100:00100000:2.0:1778628461.346107:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628461/real 1778628461] req@ffff954350197b80 x1865013828642944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628461.346108:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628461.346115:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628461.346123:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628461.346128:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828642944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628461.346136:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828643456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628461.346137:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628461.346143:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628461.346150:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828642944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628461.346172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628461.346187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628461.346204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628461/real 1778628461] req@ffff954350194e00 x1865013828643072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628461.346213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828643072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628461.346219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628461.346221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628461.346224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628461.346227:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628461.346238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628461.346244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628461/real 1778628461] req@ffff954350196a00 x1865013828643200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628461.346251:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828643200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628461.346256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628461.346257:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628461.346259:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628461.346262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628461.346272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628461.346277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628461/real 1778628461] req@ffff954350194a80 x1865013828643328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628461.346284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828643328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628461.346294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628461.346296:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628461.346298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628461.346302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628461.346314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628461.346320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628461/real 1778628461] req@ffff954350194380 x1865013828643456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628461.346327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828643456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628461.346332:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628461.346334:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628461.346336:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628461.537768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628461.537902:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628462.305780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628462.561757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628462.561843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628463.329844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628463.585756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628463.585929:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628464.353842:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628464.609906:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628464.610026:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628465.377772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628465.633775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628465.633996:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628466.401859:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628466.465912:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628466.465929:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628466.465936:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628466.465950:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12800 00000100:00080000:2.0:1778628466.465967:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628466.465977:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628466.466042:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828643584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628466.466082:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628466.466103:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628466.466106:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628466.466113:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12800 00000100:00080000:2.0:1778628466.466119:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628466.466126:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628466.466141:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628466.466160:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828643712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628466.466177:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628466.466182:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628466.466185:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628466.466194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828643584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628466.466195:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12800 00000100:00080000:2.0:1778628466.466202:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628466.466212:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628466.466220:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628466.466235:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828643840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628466.466247:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628466.466252:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628466.466254:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628466.466260:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12800 00000100:00080000:2.0:1778628466.466264:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628466.466269:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628466.466284:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628466.466299:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828643968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628466.466322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628466.466322:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628466.466333:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628466.466338:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628466.466349:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12800 00000100:00080000:2.0:1778628466.466360:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628466.466370:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628466.466382:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628466.466391:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628466/real 1778628466] req@ffff954358789c00 x1865013828643584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628466.466413:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828644096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628466.466437:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828643584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628466.466468:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628466.466476:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628466.466482:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828643584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628466.466524:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628466.466551:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628466.466565:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628466/real 1778628466] req@ffff954358789880 x1865013828643712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628466.466581:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828643712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628466.466600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628466.466616:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628466.466621:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628466.466629:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628466.466654:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628466.466666:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628466/real 1778628466] req@ffff95435878b100 x1865013828643840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628466.466749:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828643840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628466.466760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628466.466764:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628466.466768:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628466.466776:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628466.466803:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628466.466851:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628466/real 1778628466] req@ffff95435878bb80 x1865013828643968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628466.466876:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828643968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628466.466894:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628466.466901:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628466.466907:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828643968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628466.466921:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628466.466971:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628466.466987:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628466/real 1778628466] req@ffff95435878b480 x1865013828644096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628466.467002:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828644096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628466.467013:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628466.467017:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628466.467022:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628466.657869:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628466.658003:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628467.425784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778628467.682612:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628467.682623:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628468.449805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778628468.706076:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628468.706583:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628469.473742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628469.730049:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628469.730226:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628470.497793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778628470.753943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628470.754194:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628471.521829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628471.585925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628471.585941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628471.585949:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628471.585969:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12806 00000100:00080000:0.0:1778628471.585991:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628471.586004:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628471.586094:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828644224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628471.586142:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628471.586148:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628471.586150:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628471.586181:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12806 00000100:00080000:0.0:1778628471.586190:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628471.586196:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628471.586211:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628471.586223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828644224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628471.586235:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828644352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628471.586252:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628471.586256:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628471.586259:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628471.586264:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12806 00000100:00080000:0.0:1778628471.586268:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628471.586273:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628471.586279:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628471.586296:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828644480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628471.586309:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628471.586314:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628471.586316:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628471.586320:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12806 00000100:00080000:0.0:1778628471.586325:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778628471.586334:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628471.586335:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628471.586347:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628471.586362:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828644608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628471.586383:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628471.586387:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628471.586402:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628471.586409:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12806 00000100:00080000:0.0:1778628471.586415:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628471.586422:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628471.586431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628471/real 1778628471] req@ffff954342f4bb80 x1865013828644224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778628471.586433:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628471.586454:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828644736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628471.586507:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828644224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628471.586525:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628471.586533:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628471.586550:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828644224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628471.586588:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628471.586624:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628471.586641:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628471/real 1778628471] req@ffff954342f49880 x1865013828644352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628471.586659:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828644352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628471.586713:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628471.586754:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628471.586762:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628471.586776:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628471.586829:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628471.586847:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628471/real 1778628471] req@ffff954342f4a300 x1865013828644480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628471.586864:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828644480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628471.586874:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628471.586879:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628471.586883:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628471.586890:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628471.586915:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628471.586939:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628471/real 1778628471] req@ffff954342f4b480 x1865013828644608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628471.586955:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828644608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628471.586966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628471.586970:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628471.586974:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628471.586981:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628471.587003:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628471.587012:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628471/real 1778628471] req@ffff954342f49c00 x1865013828644736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628471.587026:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828644736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628471.587053:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628471.587056:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628471.587061:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628471.778099:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628471.778169:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628472.545796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628472.801790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628472.801873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628473.569907:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628473.825736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628473.826010:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628474.593907:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628474.849753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628474.849814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628475.617790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628475.873802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628475.873905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628476.641839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628476.705853:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628476.705869:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628476.705876:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628476.705892:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12811 00000100:00080000:0.0:1778628476.705910:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628476.705919:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628476.705997:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828644864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628476.706040:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628476.706046:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628476.706073:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628476.706081:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12811 00000100:00080000:0.0:1778628476.706088:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628476.706094:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628476.706118:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628476.706121:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828644864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628476.706143:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828644992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628476.706161:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628476.706166:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628476.706168:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628476.706174:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12811 00000100:00080000:0.0:1778628476.706179:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628476.706183:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628476.706191:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628476.706206:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828645120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628476.706218:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628476.706223:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628476.706227:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628476.706232:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12811 00000100:00080000:0.0:1778628476.706236:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628476.706240:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628476.706247:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778628476.706251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628476.706277:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828645248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628476.706293:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628476.706298:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628476.706301:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628476.706306:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12811 00000100:00080000:0.0:1778628476.706312:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628476.706318:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628476.706324:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628476.706344:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828645376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628476.706344:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628476/real 1778628476] req@ffff954342f48700 x1865013828644864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628476.706421:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828644864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628476.706445:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628476.706455:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628476.706465:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828644864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628476.706511:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628476.706549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628476.706571:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628476/real 1778628476] req@ffff954342f4b100 x1865013828644992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628476.706598:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828644992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628476.706638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628476.706647:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628476.706654:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828644992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628476.706667:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628476.706769:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628476.706794:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628476/real 1778628476] req@ffff954342f49f80 x1865013828645120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628476.706822:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828645120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628476.706857:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628476.706863:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628476.706867:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628476.706876:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628476.706909:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628476.706924:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628476/real 1778628476] req@ffff954342f49500 x1865013828645248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628476.706939:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828645248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628476.706951:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628476.706955:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628476.706959:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628476.706981:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628476.707004:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628476.707015:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628476/real 1778628476] req@ffff954342f4ad80 x1865013828645376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628476.707029:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828645376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628476.707041:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628476.707044:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628476.707058:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628476.897826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628476.897986:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628477.665805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628477.921803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628477.921985:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628478.689889:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628478.945782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628478.945852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628479.713735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628479.969809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628479.969900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628480.737842:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628480.993840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628480.993972:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628481.761795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628481.825786:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628481.825797:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628481.825802:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628481.825813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12816 00000100:00080000:0.0:1778628481.825827:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628481.825837:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628481.825884:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828645504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628481.825923:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628481.825947:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628481.825950:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628481.825956:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12816 00000100:00080000:0.0:1778628481.825963:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628481.825969:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628481.825981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628481.825999:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828645632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628481.826016:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628481.826023:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628481.826028:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778628481.826031:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828645504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628481.826034:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12816 00000100:00080000:0.0:1778628481.826043:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628481.826050:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628481.826058:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628481.826073:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828645760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628481.826087:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628481.826093:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628481.826095:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628481.826100:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12816 00000100:00080000:0.0:1778628481.826104:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628481.826109:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628481.826124:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628481.826142:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828645888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628481.826161:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628481.826167:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628481.826172:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778628481.826184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628481.826193:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12816 00000100:00080000:0.0:1778628481.826206:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628481.826214:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628481.826226:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628481.826251:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828646016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778628481.826273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628481/real 1778628481] req@ffff954342f48a80 x1865013828645504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628481.826319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828645504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628481.826356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628481.826366:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628481.826374:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828645504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628481.826423:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628481.826477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628481.826504:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628481/real 1778628481] req@ffff954342f4a680 x1865013828645632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628481.826563:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828645632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628481.826576:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628481.826581:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628481.826604:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628481.826615:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778628481.826650:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628481.826668:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628481/real 1778628481] req@ffff954342f4ad80 x1865013828645760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628481.826755:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828645760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628481.826767:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628481.826772:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628481.826777:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628481.826787:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628481.826823:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628481.826841:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628481/real 1778628481] req@ffff954342f49500 x1865013828645888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628481.826860:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828645888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628481.826873:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628481.826877:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628481.826882:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828645888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628481.826892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778628481.826920:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628481.826950:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628481/real 1778628481] req@ffff954342f49f80 x1865013828646016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628481.826968:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828646016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628481.826981:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628481.826986:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628481.826991:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628482.017824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628482.017927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628482.785813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628483.041841:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628483.041875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628483.809756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628484.065762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628484.065834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628484.833783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628485.089812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628485.089917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628485.857804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628486.113868:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628486.113984:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628486.881783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628486.945898:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628486.945915:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628486.945922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628486.945937:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12821 00000100:00080000:2.0:1778628486.945956:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628486.945965:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628486.946026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828646144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628486.946076:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628486.946082:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628486.946085:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628486.946091:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12821 00000100:00080000:2.0:1778628486.946096:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628486.946125:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628486.946141:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628486.946161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828646144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628486.946165:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828646272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628486.946199:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628486.946205:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628486.946208:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628486.946216:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12821 00000100:00080000:2.0:1778628486.946222:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628486.946226:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628486.946233:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628486.946249:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828646400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628486.946262:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628486.946267:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628486.946270:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628486.946275:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12821 00000100:00080000:2.0:1778628486.946279:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628486.946283:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628486.946289:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628486.946301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628486.946312:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828646528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628486.946328:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628486.946340:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628486.946343:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628486.946353:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12821 00000100:00080000:2.0:1778628486.946363:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628486.946373:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628486.946382:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628486/real 1778628486] req@ffff954358788380 x1865013828646144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778628486.946388:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628486.946441:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828646656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628486.946459:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828646144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628486.946477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628486.946501:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628486.946507:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828646144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628486.946543:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628486.946580:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628486.946597:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628486/real 1778628486] req@ffff95435878aa00 x1865013828646272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628486.946613:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828646272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628486.946624:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628486.946628:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628486.946632:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628486.946640:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628486.946659:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628486.946706:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628486/real 1778628486] req@ffff95435878b480 x1865013828646400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628486.946723:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828646400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628486.946733:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628486.946737:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628486.946741:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628486.946749:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628486.946789:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628486.946804:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628486/real 1778628486] req@ffff95435878bb80 x1865013828646528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628486.946819:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828646528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628486.946829:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628486.946834:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628486.946837:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628486.946845:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628486.946863:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628486.946873:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628486/real 1778628486] req@ffff95435878b100 x1865013828646656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628486.946887:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828646656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628486.946897:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628486.946901:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628486.946904:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628487.137808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628487.137980:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628487.905860:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628488.161828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628488.162017:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628488.929798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628489.185835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628489.185977:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628489.953812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628490.209870:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628490.209951:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628490.977767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628491.233823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628491.233934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628492.001804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628492.065823:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628492.065833:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628492.065838:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628492.065848:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12826 00000100:00080000:1.0:1778628492.065862:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628492.065869:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628492.065917:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828646784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628492.065946:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628492.065949:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628492.065950:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628492.065954:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12826 00000100:00080000:1.0:1778628492.065956:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628492.065959:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628492.065969:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628492.065980:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828646912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628492.066000:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828646784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628492.066008:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628492.066016:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628492.066018:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628492.066022:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12826 00000100:00080000:1.0:1778628492.066025:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628492.066028:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628492.066032:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628492.066045:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828647040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628492.066052:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628492.066055:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628492.066056:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628492.066059:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12826 00000100:00080000:1.0:1778628492.066078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628492.066081:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628492.066084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628492.066091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628492.066094:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828647168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628492.066101:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628492.066104:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628492.066105:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628492.066109:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12826 00000100:00080000:1.0:1778628492.066118:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628492.066122:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628492.066126:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628492.066135:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828647296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628492.066140:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628492/real 1778628492] req@ffff954350194000 x1865013828646784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628492.066161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828646784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628492.066172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628492.066176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628492.066180:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828646784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628492.066205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628492.066221:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628492.066229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628492/real 1778628492] req@ffff954350197480 x1865013828646912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628492.066238:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828646912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628492.066244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628492.066246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628492.066248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828646912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628492.066252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628492.066262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628492.066267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628492/real 1778628492] req@ffff954350196d80 x1865013828647040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628492.066286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828647040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628492.066292:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628492.066294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628492.066296:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628492.066300:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628492.066328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628492.066336:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628492/real 1778628492] req@ffff954350197800 x1865013828647168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628492.066345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828647168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628492.066351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628492.066353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628492.066355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628492.066359:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628492.066370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628492.066376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628492/real 1778628492] req@ffff954350194700 x1865013828647296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628492.066383:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828647296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628492.066403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628492.066407:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628492.066410:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628492.257788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628492.257930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628493.025854:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628493.281816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628493.281974:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628494.049808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628494.305855:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628494.305946:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628495.073812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628495.329784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628495.329941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628496.097820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628496.353799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628496.354009:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628497.121809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628497.185803:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628497.185817:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628497.185822:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628497.185835:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12831 00000100:00080000:1.0:1778628497.185849:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628497.185859:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628497.185904:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828647424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628497.185941:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628497.185946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628497.185949:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628497.185954:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12831 00000100:00080000:1.0:1778628497.185958:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628497.185963:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628497.185975:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628497.185993:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828647552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628497.186001:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828647424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628497.186023:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628497.186041:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628497.186044:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628497.186052:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12831 00000100:00080000:1.0:1778628497.186058:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628497.186065:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628497.186072:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628497.186089:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828647680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628497.186103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628497.186116:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628497.186122:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628497.186124:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628497.186131:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12831 00000100:00080000:1.0:1778628497.186138:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628497.186143:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628497.186150:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628497.186168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628497/real 1778628497] req@ffff954350197100 x1865013828647424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628497.186168:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828647808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628497.186242:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628497.186249:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628497.186252:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628497.186259:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12831 00000100:00080000:1.0:1778628497.186268:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628497.186269:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828647424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628497.186281:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778628497.186293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778628497.186302:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778628497.186305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628497.186314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828647424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628497.186329:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828647936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628497.186353:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628497.186393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628497.186410:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628497/real 1778628497] req@ffff954350194700 x1865013828647552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628497.186426:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828647552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628497.186452:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628497.186457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628497.186462:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628497.186470:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628497.186495:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628497.186508:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628497/real 1778628497] req@ffff954350197800 x1865013828647680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628497.186522:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828647680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628497.186531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628497.186535:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628497.186539:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628497.186546:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628497.186564:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628497.186575:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628497/real 1778628497] req@ffff954350196d80 x1865013828647808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628497.186589:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828647808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628497.186598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628497.186601:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628497.186617:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628497.186629:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628497.186710:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628497.186736:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628497/real 1778628497] req@ffff954350197480 x1865013828647936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628497.186755:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828647936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628497.186766:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628497.186770:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628497.186774:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828647936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628497.377744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628497.377915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628498.145858:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628498.401849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628498.401929:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628499.169821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628499.425771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628499.425927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628500.193761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628500.449851:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628500.449950:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628501.217813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628501.473856:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628501.473922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628502.241784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628502.305819:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628502.305832:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628502.305838:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628502.305865:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12836 00000100:00080000:2.0:1778628502.305906:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628502.305920:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628502.305971:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828648064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628502.306012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628502.306017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628502.306020:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628502.306025:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12836 00000100:00080000:2.0:1778628502.306031:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628502.306036:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628502.306050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628502.306070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828648192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628502.306088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628502.306096:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828648064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628502.306098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628502.306133:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628502.306154:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12836 00000100:00080000:2.0:1778628502.306161:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628502.306167:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628502.306175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628502.306195:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828648320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628502.306211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628502.306226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628502.306230:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628502.306236:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12836 00000100:00080000:2.0:1778628502.306243:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628502.306249:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628502.306257:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628502.306275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828648448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628502.306289:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628502.306295:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628502.306299:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628502.306308:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648192:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628502.306308:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12836 00000100:00080000:2.0:1778628502.306320:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628502.306332:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628502.306339:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628502.306374:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828648576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628502.306403:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628502.306462:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628502.306521:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628502.497804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628502.497971:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628503.265797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628503.521789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628503.521877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628504.289855:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628504.545781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628504.545855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628505.313775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778628505.378009:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778628505.378084:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778628505.378107:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628505.378159:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628505.378176:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628505.378191:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628505.378206:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628505.378222:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628502/real 1778628505] req@ffff954358789c00 x1865013828648192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628505.378266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828648192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628505.378288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628505.378298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628505.378306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628505.378319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628502/real 1778628505] req@ffff954358788e00 x1865013828648320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628505.378345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828648320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628505.378363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628505.378370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628505.378378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628505.378390:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628502/real 1778628505] req@ffff954358788700 x1865013828648448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628505.378438:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828648448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628505.378454:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628505.378460:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628505.378487:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628505.378501:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628502/real 1778628505] req@ffff954358789880 x1865013828648064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628505.378528:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828648064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628505.378545:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628505.378552:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628505.378557:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828648064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628505.378611:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628502/real 1778628505] req@ffff954358789500 x1865013828648576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628505.378637:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828648576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628505.378653:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628505.378659:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628505.378665:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628505.569752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628505.569829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628506.337765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628506.593759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628506.593888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628507.361737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628507.425740:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628507.425747:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628507.425762:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628507.425770:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12841 00000100:00080000:2.0:1778628507.425809:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628507.425819:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628507.425874:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828648704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628507.425902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628507.425906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628507.425908:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628507.425914:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12841 00000100:00080000:2.0:1778628507.425919:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628507.425924:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628507.425937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628507.425956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828648832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628507.425976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628507.425982:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828648704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628507.426025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628507.426027:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628507.426032:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12841 00000100:00080000:2.0:1778628507.426036:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628507.426040:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628507.426045:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628507.426060:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828648960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628507.426085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628507.426085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628507.426090:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628507.426092:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628507.426099:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12841 00000100:00080000:2.0:1778628507.426105:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628507.426110:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628507.426116:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628507.426126:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628507/real 1778628507] req@ffff95435878b800 x1865013828648704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628507.426134:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828649088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628507.426150:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628507.426153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628507.426155:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628507.426160:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12841 00000100:00100000:0.0:1778628507.426161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828648704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628507.426164:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628507.426168:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628507.426176:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778628507.426186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628507.426194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628507.426196:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828649216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628507.426201:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828648704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628507.426233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628507.426260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628507.426273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628507/real 1778628507] req@ffff95435878a300 x1865013828648832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628507.426287:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828648832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628507.426296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628507.426299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628507.426303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628507.426309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628507.426339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628507.426351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628507/real 1778628507] req@ffff95435878ad80 x1865013828648960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628507.426363:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828648960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628507.426372:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628507.426375:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628507.426378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828648960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628507.426398:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628507.426421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628507.426431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628507/real 1778628507] req@ffff954358789180 x1865013828649088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628507.426441:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828649088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628507.426448:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628507.426452:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628507.426455:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628507.426460:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628507.426477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628507.426487:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628507/real 1778628507] req@ffff954358788a80 x1865013828649216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628507.426498:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828649216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628507.426506:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628507.426510:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628507.426513:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628507.617754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628507.617886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628508.385753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628508.641819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628508.641910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628509.409744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628509.665774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628509.665862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628510.433789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628510.689805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628510.689878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628511.457780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628511.713775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628511.714010:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628512.481773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628512.545807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628512.545819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628512.545825:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628512.545839:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12847 00000100:00080000:2.0:1778628512.545855:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628512.545864:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628512.545947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828649344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628512.545987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628512.545992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628512.545994:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628512.545999:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12847 00000100:00080000:2.0:1778628512.546004:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628512.546008:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628512.546020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628512.546037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828649472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628512.546049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628512.546054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628512.546055:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628512.546062:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828649344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628512.546064:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12847 00000100:00080000:2.0:1778628512.546071:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628512.546077:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628512.546082:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628512.546123:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828649600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628512.546133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628512.546137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628512.546139:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628512.546143:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12847 00000100:00080000:2.0:1778628512.546149:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628512.546153:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628512.546161:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628512.546166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628512.546180:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828649728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628512.546192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628512.546196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628512.546198:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628512.546201:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12847 00000100:00080000:2.0:1778628512.546205:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628512.546209:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628512.546215:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628512.546226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628512/real 1778628512] req@ffff95435878a680 x1865013828649344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628512.546229:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828649856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628512.546293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828649344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628512.546314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628512.546322:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628512.546329:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828649344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628512.546366:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628512.546404:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628512.546424:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628512/real 1778628512] req@ffff954358789f80 x1865013828649472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628512.546451:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828649472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628512.546460:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628512.546464:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628512.546467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628512.546474:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628512.546496:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628512.546506:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628512/real 1778628512] req@ffff954358788a80 x1865013828649600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628512.546532:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828649600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628512.546551:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628512.546556:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628512.546561:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628512.546572:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628512.546605:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628512.546622:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628512/real 1778628512] req@ffff954358789180 x1865013828649728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628512.546643:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828649728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628512.546657:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628512.546660:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628512.546664:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628512.546718:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628512.546743:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628512.546755:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628512/real 1778628512] req@ffff95435878ad80 x1865013828649856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628512.546767:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828649856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628512.546791:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628512.546794:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628512.546798:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828649856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628512.737802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628512.737978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628513.505742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628513.761834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628513.761977:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628514.529831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628514.785769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628514.785968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628515.553840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628515.809772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628515.809986:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628516.577919:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628516.833812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628516.833935:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628517.601840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628517.665862:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628517.665876:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628517.665882:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628517.665896:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12852 00000100:00080000:2.0:1778628517.665910:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628517.665919:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628517.665961:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828649984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628517.665997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628517.666001:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628517.666003:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628517.666008:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12852 00000100:00080000:2.0:1778628517.666013:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628517.666019:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628517.666032:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628517.666076:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828650112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628517.666093:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628517.666098:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628517.666102:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628517.666113:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828649984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628517.666114:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12852 00000100:00080000:2.0:1778628517.666128:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628517.666134:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628517.666177:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628517.666199:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828650240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628517.666212:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628517.666215:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628517.666217:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628517.666222:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12852 00000100:00080000:2.0:1778628517.666228:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628517.666250:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628517.666255:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628517.666267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628517.666273:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828650368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628517.666282:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628517.666286:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628517.666288:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628517.666293:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12852 00000100:00080000:2.0:1778628517.666299:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628517.666303:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628517.666310:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628517.666318:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828650496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628517.666344:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628517/real 1778628517] req@ffff95435878a300 x1865013828649984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628517.666372:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828649984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628517.666388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628517.666394:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628517.666399:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828649984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628517.666443:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628517.666471:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628517.666485:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628517/real 1778628517] req@ffff95435878b800 x1865013828650112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628517.666500:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828650112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628517.666532:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628517.666537:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628517.666542:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628517.666560:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628517.666586:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628517.666597:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628517/real 1778628517] req@ffff954358789500 x1865013828650240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628517.666611:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828650240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628517.666621:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628517.666625:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628517.666628:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628517.666635:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628517.666656:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628517.666666:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628517/real 1778628517] req@ffff954358789880 x1865013828650368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628517.666729:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828650368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628517.666741:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628517.666747:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628517.666768:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628517.666781:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628517.666846:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628517.666870:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628517/real 1778628517] req@ffff954358788700 x1865013828650496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628517.666896:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828650496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628517.666911:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628517.666916:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628517.666920:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628517.857778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628517.858044:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628518.625747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628518.881769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628518.881871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628519.649881:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628519.905793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628519.906003:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628520.673802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628520.929842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628520.929962:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628521.697814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628521.953801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628521.953904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628522.721773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628522.785859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628522.785871:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628522.785877:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628522.785890:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12857 00000100:00080000:1.0:1778628522.785902:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628522.785920:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628522.785965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828650624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628522.786003:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628522.786008:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628522.786011:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628522.786034:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12857 00000100:00080000:1.0:1778628522.786038:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628522.786043:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628522.786052:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628522.786059:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828650624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628522.786067:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828650752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628522.786077:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628522.786080:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628522.786081:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628522.786084:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12857 00000100:00080000:1.0:1778628522.786112:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628522.786116:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628522.786121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778628522.786124:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628522.786139:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828650880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628522.786147:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628522.786149:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628522.786151:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628522.786155:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12857 00000100:00100000:0.0:1778628522.786159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628522/real 1778628522] req@ffff954350194000 x1865013828650624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628522.786160:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628522.786169:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628522.786177:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628522.786184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828650624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628522.786186:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828651008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628522.786214:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628522.786215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628522.786220:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628522.786223:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628522.786227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778628522.786229:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12857 00000100:00080000:1.0:1778628522.786232:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628522.786234:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628522.786237:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628522.786247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828650624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628522.786249:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828651136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628522.786284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628522.786307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628522.786317:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628522/real 1778628522] req@ffff954350194380 x1865013828650752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628522.786327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828650752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628522.786333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628522.786336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628522.786339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628522.786343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628522.786354:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628522.786360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628522/real 1778628522] req@ffff954350194a80 x1865013828650880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628522.786368:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828650880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628522.786374:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628522.786376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628522.786377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828650880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628522.786382:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628522.786392:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628522.786406:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628522/real 1778628522] req@ffff954350196a00 x1865013828651008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628522.786414:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828651008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628522.786420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628522.786423:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628522.786425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628522.786429:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628522.786440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628522.786445:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628522/real 1778628522] req@ffff954350194e00 x1865013828651136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628522.786453:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828651136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628522.786459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628522.786461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628522.786463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628522.977792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628522.977851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628523.745818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628524.001808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628524.001927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628524.769790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628525.025819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628525.025926:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628525.793799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628526.049820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628526.049949:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628526.817803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628527.073843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628527.073958:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628527.841775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628527.905787:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628527.905799:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628527.905805:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628527.905817:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12862 00000100:00080000:1.0:1778628527.905831:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628527.905840:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628527.905884:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828651264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628527.905932:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628527.905938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628527.905940:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628527.905946:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12862 00000100:00080000:1.0:1778628527.905950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628527.905955:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628527.905967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628527.905983:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828651392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628527.905996:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628527.906000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628527.906002:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628527.906007:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12862 00000100:00080000:1.0:1778628527.906010:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628527.906014:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628527.906035:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628527.906050:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828651520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628527.906066:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628527.906076:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1778628527.906079:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828651264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628527.906080:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628527.906099:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12862 00000100:00080000:1.0:1778628527.906114:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628527.906121:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628527.906129:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628527.906145:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828651648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628527.906158:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628527.906163:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628527.906165:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628527.906170:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12862 00000100:00080000:1.0:1778628527.906197:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628527.906202:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628527.906207:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778628527.906217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628527.906227:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828651776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628527.906307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628527/real 1778628527] req@ffff954350197b80 x1865013828651264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628527.906331:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828651264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628527.906344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628527.906349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628527.906354:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828651264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628527.906362:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628527.906387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628527.906398:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628527/real 1778628527] req@ffff954350195180 x1865013828651392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628527.906410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828651392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628527.906418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628527.906421:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628527.906423:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628527.906452:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628527.906466:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628527.906474:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628527/real 1778628527] req@ffff954350195c00 x1865013828651520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628527.906485:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828651520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628527.906509:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628527.906513:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628527.906516:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628527.906522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628527.906542:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628527.906550:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628527/real 1778628527] req@ffff954350195500 x1865013828651648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628527.906560:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828651648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628527.906568:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628527.906570:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628527.906573:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628527.906578:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628527.906591:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628527.906599:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628527/real 1778628527] req@ffff954350195f80 x1865013828651776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628527.906635:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828651776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628527.906644:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628527.906647:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628527.906657:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828651776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628528.097772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628528.097925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628528.865799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628529.121788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628529.121918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628529.889750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628530.145766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628530.145968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628530.913772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628531.169777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628531.169967:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628531.937793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628532.193810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628532.193907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628532.961772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628533.025792:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628533.025804:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628533.025828:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628533.025840:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12867 00000100:00080000:1.0:1778628533.025854:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628533.025865:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628533.025915:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828651904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628533.025968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628533.025976:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628533.025980:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628533.025989:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12867 00000100:00080000:1.0:1778628533.025995:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628533.026000:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628533.026022:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628533.026043:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828652032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628533.026058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628533.026064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628533.026067:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628533.026072:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12867 00000100:00080000:1.0:1778628533.026077:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628533.026081:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628533.026086:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628533.026096:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828652160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628533.026124:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828651904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628533.026134:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628533.026143:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628533.026147:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628533.026158:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12867 00000100:00080000:1.0:1778628533.026168:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628533.026176:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628533.026188:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628533.026214:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828652288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628533.026235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628533.026246:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628533.026256:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628533.026260:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628533.026269:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12867 00000100:00080000:1.0:1778628533.026277:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628533.026294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628533/real 1778628533] req@ffff954350195880 x1865013828651904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628533.026305:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628533.026327:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628533.026333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828651904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628533.026353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628533.026360:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628533.026369:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828652416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628533.026370:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828651904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628533.026391:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628533.026455:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628533.026473:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628533/real 1778628533] req@ffff954350195f80 x1865013828652032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628533.026489:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828652032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628533.026528:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628533.026534:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628533.026552:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628533.026584:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628533.026613:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628533.026627:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628533/real 1778628533] req@ffff954350195500 x1865013828652160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628533.026655:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828652160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628533.026666:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628533.026703:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628533.026708:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628533.026716:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628533.026740:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628533.026754:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628533/real 1778628533] req@ffff954350195c00 x1865013828652288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628533.026768:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828652288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628533.026778:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628533.026782:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628533.026785:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628533.026792:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628533.026810:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628533.026834:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628533/real 1778628533] req@ffff954350195180 x1865013828652416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628533.026849:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828652416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628533.026860:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628533.026863:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628533.026867:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628533.217780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628533.217924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628533.985773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628534.241882:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628534.242612:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628535.009770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628535.265905:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628535.266374:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628536.033790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778628536.290193:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628536.290584:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628537.057792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628537.314480:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628537.314724:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628538.081791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628538.145874:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628538.145887:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628538.145893:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628538.145905:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12872 00000100:00080000:2.0:1778628538.145920:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628538.145930:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628538.145990:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828652544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628538.146044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628538.146052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628538.146055:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628538.146089:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12872 00000100:00080000:2.0:1778628538.146099:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628538.146108:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628538.146125:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628538.146149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828652544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628538.146157:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828652672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628538.146187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628538.146193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628538.146196:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628538.146203:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12872 00000100:00080000:2.0:1778628538.146209:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628538.146214:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628538.146222:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628538.146244:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828652800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628538.146257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628538.146263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628538.146265:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628538.146270:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12872 00000100:00080000:2.0:1778628538.146274:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628538.146278:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628538.146287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778628538.146288:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628538.146316:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828652928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628538.146341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628538.146347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628538.146350:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628538.146357:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12872 00000100:00080000:2.0:1778628538.146363:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628538.146366:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628538/real 1778628538] req@ffff954358788e00 x1865013828652544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628538.146374:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628538.146396:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628538.146448:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828652544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628538.146452:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828653056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628538.146510:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628538.146523:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628538.146535:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828652544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628538.146581:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628538.146618:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628538.146653:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628538/real 1778628538] req@ffff954358789c00 x1865013828652672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628538.146713:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828652672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628538.146726:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628538.146731:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628538.146735:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628538.146745:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628538.146774:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628538.146788:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628538/real 1778628538] req@ffff95435878b100 x1865013828652800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628538.146803:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828652800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628538.146813:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628538.146818:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628538.146821:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628538.146828:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628538.146849:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628538.146859:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628538/real 1778628538] req@ffff95435878bb80 x1865013828652928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628538.146873:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828652928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628538.146892:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628538.146897:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628538.146901:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828652928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628538.146909:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628538.146935:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628538.146946:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628538/real 1778628538] req@ffff95435878b480 x1865013828653056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628538.146960:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828653056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628538.146971:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628538.146974:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628538.146978:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628538.337879:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628538.337909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628539.105803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628539.361880:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628539.362178:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628540.129782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628540.385780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628540.385894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628541.153809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628541.409815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628541.409899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628542.177742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628542.433784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628542.433818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628543.201793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628543.265889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628543.265901:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628543.265907:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628543.265919:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12877 00000100:00080000:1.0:1778628543.265934:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628543.265941:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628543.265990:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828653184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628543.266024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628543.266029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628543.266031:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628543.266036:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12877 00000100:00080000:1.0:1778628543.266040:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628543.266044:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628543.266056:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628543.266069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828653312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628543.266079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628543.266083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628543.266085:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628543.266086:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828653184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628543.266088:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12877 00000100:00080000:1.0:1778628543.266093:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628543.266097:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628543.266101:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628543.266139:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828653440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628543.266149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628543.266153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628543.266155:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628543.266159:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12877 00000100:00080000:1.0:1778628543.266163:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628543.266168:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628543.266174:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628543.266204:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828653568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628543.266219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628543.266225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628543.266228:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628543.266235:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12877 00000800:00000100:0.0:1778628543.266240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628543.266241:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628543.266248:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628543.266256:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628543.266323:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828653696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628543.266360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628543/real 1778628543] req@ffff954350197b80 x1865013828653184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628543.266385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828653184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628543.266399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628543.266406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628543.266411:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828653184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628543.266460:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628543.266490:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628543.266505:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628543/real 1778628543] req@ffff954350194e00 x1865013828653312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628543.266518:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828653312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628543.266527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628543.266531:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628543.266534:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628543.266540:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628543.266556:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628543.266564:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628543/real 1778628543] req@ffff954350196a00 x1865013828653440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628543.266595:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828653440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628543.266604:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628543.266607:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628543.266629:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628543.266637:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628543.266661:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628543.266673:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628543/real 1778628543] req@ffff954350194a80 x1865013828653568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628543.266700:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828653568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628543.266706:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628543.266708:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628543.266710:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628543.266714:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628543.266726:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628543.266732:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628543/real 1778628543] req@ffff954350194380 x1865013828653696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628543.266840:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828653696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628543.266856:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628543.266859:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628543.266862:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628543.457832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628543.457900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628544.225768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628544.481839:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628544.481870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628545.249782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628545.505806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628545.505856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628546.273815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628546.529843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628546.529875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628547.297785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628547.553801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628547.553864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628548.321779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628548.385887:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628548.385904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628548.385911:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628548.385926:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12882 00000100:00080000:2.0:1778628548.385963:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628548.385974:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628548.386036:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828653824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628548.386086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628548.386091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628548.386094:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628548.386100:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12882 00000100:00080000:2.0:1778628548.386104:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628548.386109:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628548.386149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628548.386173:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828653952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628548.386222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628548.386232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778628548.386241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828653824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628548.386242:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628548.386263:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12882 00000100:00080000:2.0:1778628548.386273:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628548.386284:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628548.386297:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628548.386324:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828654080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628548.386358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778628548.386369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628548.386375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628548.386380:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628548.386389:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12882 00000100:00080000:2.0:1778628548.386400:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628548.386409:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628548.386422:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628548.386428:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628548/real 1778628548] req@ffff95435878aa00 x1865013828653824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628548.386456:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828654208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628548.386459:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828653824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628548.386519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628548.386546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628548.386554:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628548.386569:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628548.386571:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628548.386587:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12882 00000100:00100000:0.0:1778628548.386589:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828653824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628548.386601:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628548.386612:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628548.386649:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653952:192.168.204.102@tcp:38:kworker.0 00000008:00100000:2.0:1778628548.386666:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778628548.386757:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628548.386813:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628548/real 1778628548] req@ffff954358788380 x1865013828653952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628548.386822:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828654336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628548.386859:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828653952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628548.386881:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628548.386887:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628548.386892:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828653952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628548.386903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628548.386943:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628548.386955:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628548/real 1778628548] req@ffff95435878b480 x1865013828654080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628548.386969:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828654080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628548.386979:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628548.386983:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628548.386986:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628548.386994:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628548.387012:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628548.387023:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628548/real 1778628548] req@ffff95435878bb80 x1865013828654208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628548.387036:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828654208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628548.387054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628548.387059:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628548.387063:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628548.387088:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628548.387112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628548.387124:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628548/real 1778628548] req@ffff95435878b100 x1865013828654336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628548.387138:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828654336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628548.387148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628548.387152:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628548.387156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628548.577815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628548.577872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628549.345773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628549.601830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628549.601875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628550.369844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628550.625790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628550.625829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628551.393796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628551.649817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628551.649830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628552.417777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628552.673824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628552.673827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628553.441892:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628553.505839:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628553.505871:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628553.505880:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628553.505895:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12888 00000100:00080000:3.0:1778628553.505912:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628553.505922:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628553.505977:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828654464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628553.506017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628553.506022:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628553.506024:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628553.506030:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12888 00000100:00080000:3.0:1778628553.506034:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628553.506047:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628553.506061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628553.506080:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828654592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628553.506081:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828654464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628553.506126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628553.506145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628553.506148:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628553.506154:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12888 00000100:00080000:3.0:1778628553.506160:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628553.506165:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628553.506173:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628553.506195:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828654720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628553.506211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628553.506223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628553.506230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628553.506234:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628553.506244:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12888 00000100:00080000:3.0:1778628553.506252:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628553.506260:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628553.506270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628553.506273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628553/real 1778628553] req@ffff95434701f100 x1865013828654464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628553.506299:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828654848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628553.506342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628553.506346:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628553.506349:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628553.506359:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12888 00000100:00100000:0.0:1778628553.506360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828654464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628553.506366:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628553.506378:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778628553.506385:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628553.506394:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628553.506402:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828654464:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778628553.506403:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628553.506427:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828654976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628553.506447:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628553.506479:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628553.506494:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628553/real 1778628553] req@ffff95434701c380 x1865013828654592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628553.506508:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828654592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628553.506518:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628553.506522:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628553.506525:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628553.506531:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628553.506547:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628553.506556:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628553/real 1778628553] req@ffff95434701d500 x1865013828654720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628553.506568:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828654720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628553.506592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628553.506596:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628553.506599:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628553.506606:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628553.506629:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628553.506639:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628553/real 1778628553] req@ffff95434701f800 x1865013828654848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628553.506651:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828654848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628553.506659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628553.506662:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628553.506665:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628553.506709:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628553.506730:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628553.506742:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628553/real 1778628553] req@ffff95434701c700 x1865013828654976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628553.506755:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828654976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628553.506764:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628553.506768:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628553.506779:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828654976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628553.697835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628553.697840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628554.465818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628554.721819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628554.721867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628555.489786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628555.745799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628555.745920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628556.513780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628556.769770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628556.769879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628557.537813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628557.793797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628557.793941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628558.561761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628558.625763:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628558.625773:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628558.625778:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628558.625790:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12893 00000100:00080000:3.0:1778628558.625804:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628558.625814:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628558.625862:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828655104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628558.625898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628558.625903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628558.625906:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628558.625911:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12893 00000100:00080000:3.0:1778628558.625915:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628558.625919:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628558.625930:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628558.625944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828655232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628558.625957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628558.625962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628558.625964:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628558.625970:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12893 00000100:00080000:3.0:1778628558.626014:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628558.626024:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628558.626036:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628558.626037:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828655104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628558.626069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828655360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628558.626094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628558.626101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628558.626104:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628558.626112:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12893 00000100:00080000:3.0:1778628558.626121:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628558.626132:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628558.626133:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778628558.626148:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628558.626170:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828655488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628558.626184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628558.626189:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628558/real 1778628558] req@ffff95434701c000 x1865013828655104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628558.626191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628558.626197:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628558.626211:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12893 00000100:00080000:3.0:1778628558.626222:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628558.626228:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628558.626234:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628558.626236:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828655104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628558.626253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778628558.626253:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828655616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628558.626270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628558.626282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828655104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628558.626299:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628558.626336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628558.626354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628558/real 1778628558] req@ffff95434701ea00 x1865013828655232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628558.626370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828655232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628558.626380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628558.626403:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628558.626429:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628558.626469:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628558.626511:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628558.626533:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628558/real 1778628558] req@ffff95434701d880 x1865013828655360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628558.626558:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828655360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628558.626571:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628558.626576:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628558.626580:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628558.626589:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628558.626616:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628558.626630:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628558/real 1778628558] req@ffff95434701d180 x1865013828655488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628558.626644:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828655488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628558.626654:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628558.626658:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628558.626661:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628558.626720:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628558.626746:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628558.626773:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628558/real 1778628558] req@ffff95434701f480 x1865013828655616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628558.626788:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828655616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628558.626799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628558.626803:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628558.626806:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628558.817773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628558.817824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628559.585801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628559.841814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628559.841862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628560.609775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628560.865791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628560.865852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628561.633804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628561.889797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628561.889841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628562.657782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628562.913785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628562.913814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628563.681777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628563.745802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628563.745815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628563.745821:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628563.745833:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12898 00000100:00080000:2.0:1778628563.745849:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628563.745859:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628563.745923:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828655744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628563.745980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628563.745985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628563.745988:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628563.745995:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12898 00000100:00080000:2.0:1778628563.746002:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628563.746007:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628563.746020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628563.746042:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828655872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628563.746058:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828655744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628563.746061:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628563.746075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628563.746080:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628563.746087:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12898 00000100:00080000:2.0:1778628563.746091:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628563.746095:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628563.746101:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628563.746118:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828656000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628563.746131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628563.746136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628563.746138:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628563.746143:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12898 00000100:00080000:2.0:1778628563.746158:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628563.746164:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628563.746171:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628563.746182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628563.746195:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828656128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628563.746210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628563.746229:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628563.746234:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628563.746244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628563/real 1778628563] req@ffff954358789c00 x1865013828655744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628563.746246:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12898 00000100:00080000:2.0:1778628563.746260:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628563.746275:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628563.746289:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628563.746304:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828655744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628563.746317:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828656256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628563.746359:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628563.746370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628563.746377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828655744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628563.746411:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628563.746446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628563.746477:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628563/real 1778628563] req@ffff954358788e00 x1865013828655872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628563.746495:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828655872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628563.746506:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628563.746510:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628563.746515:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828655872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628563.746522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628563.746551:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628563.746570:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628563/real 1778628563] req@ffff954358788700 x1865013828656000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628563.746593:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828656000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628563.746609:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628563.746616:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628563.746623:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628563.746636:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628563.746714:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628563.746732:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628563/real 1778628563] req@ffff954358789880 x1865013828656128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628563.746747:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828656128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628563.746768:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628563.746772:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628563.746777:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628563.746785:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628563.746813:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628563.746825:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628563/real 1778628563] req@ffff954358789500 x1865013828656256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628563.746839:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828656256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628563.746849:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628563.746853:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628563.746856:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628563.937804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628563.937839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628564.705785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628564.961859:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628564.961936:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628565.729766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628565.985797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628565.985930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628566.753772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628567.009783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628567.009891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628567.777768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628568.033792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628568.033969:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628568.801761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628568.865886:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628568.865902:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628568.865909:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628568.865925:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12903 00000100:00080000:1.0:1778628568.865944:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628568.865954:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628568.866019:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828656384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628568.866068:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628568.866103:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628568.866107:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628568.866118:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12903 00000100:00080000:1.0:1778628568.866127:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628568.866135:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628568.866144:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828656384:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778628568.866153:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628568.866176:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828656512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628568.866213:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628568.866220:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628568.866223:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628568.866230:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12903 00000100:00080000:1.0:1778628568.866247:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628568.866253:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628568.866262:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628568.866281:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828656640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628568.866298:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628568.866306:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628568.866310:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628568.866320:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12903 00000100:00080000:1.0:1778628568.866328:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628568.866333:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656512:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628568.866336:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628568.866358:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628568.866391:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828656768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628568.866395:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656640:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628568.866415:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628568.866421:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628568.866424:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628568.866433:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12903 00000100:00080000:1.0:1778628568.866440:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628568.866444:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656768:192.168.204.102@tcp:8:kworker.0 00000100:00080000:1.0:1778628568.866448:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628568.866457:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628568.866478:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828656896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628568.866517:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628569.057871:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628569.057918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628569.825756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628570.081793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628570.081826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628570.849769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628571.105802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628571.105886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628571.873762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778628571.937980:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778628571.938027:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778628571.938046:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628571.938108:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628571.938123:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628571.938138:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628571.938154:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628571.938190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628568/real 1778628571] req@ffff954350194700 x1865013828656896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628571.938223:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828656896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628571.938243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628571.938261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628571.938268:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628571.938281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628568/real 1778628571] req@ffff954350197800 x1865013828656768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628571.938304:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828656768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628571.938317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628571.938323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628571.938328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628571.938339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628568/real 1778628571] req@ffff954350197480 x1865013828656512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628571.938359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828656512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628571.938373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628571.938392:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628571.938398:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628571.938409:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628568/real 1778628571] req@ffff954350196d80 x1865013828656640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628571.938431:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828656640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628571.938444:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628571.938449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628571.938454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828656640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628571.938464:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628568/real 1778628571] req@ffff954350194000 x1865013828656384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628571.938485:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828656384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628571.938498:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628571.938503:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628571.938507:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828656384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628572.129839:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628572.129967:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628572.897819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628573.153788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628573.153843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628573.921733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628573.985836:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628573.985847:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628573.985853:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628573.985865:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12908 00000100:00080000:3.0:1778628573.985880:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628573.985890:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628573.985944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828657024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628573.985979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628573.985984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628573.985986:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628573.985992:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12908 00000100:00080000:3.0:1778628573.985996:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628573.986000:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628573.986012:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628573.986029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828657152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628573.986055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628573.986059:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1778628573.986062:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828657024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628573.986062:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628573.986095:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12908 00000100:00080000:3.0:1778628573.986104:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628573.986109:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628573.986117:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628573.986131:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828657280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628573.986138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628573.986148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628573.986153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628573.986155:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628573.986161:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12908 00000100:00080000:3.0:1778628573.986165:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628573.986170:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628573.986176:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628573.986180:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628573/real 1778628573] req@ffff95434701df80 x1865013828657024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628573.986193:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828657408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628573.986205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828657024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628573.986205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628573.986209:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628573.986211:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628573.986219:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12908 00000100:00080000:3.0:1778628573.986227:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628573.986232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778628573.986233:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:0.0:1778628573.986241:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:3.0:1778628573.986242:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628573.986248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828657024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628573.986261:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828657536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628573.986276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628573.986317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628573.986332:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628573/real 1778628573] req@ffff95434701e680 x1865013828657152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628573.986344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828657152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628573.986353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628573.986356:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628573.986359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628573.986364:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628573.986383:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628573.986394:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628573/real 1778628573] req@ffff95434701ca80 x1865013828657280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628573.986405:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828657280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628573.986413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628573.986415:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628573.986418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628573.986432:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628573.986454:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628573.986465:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628573/real 1778628573] req@ffff95434701f480 x1865013828657408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628573.986478:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828657408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628573.986486:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628573.986489:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628573.986492:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628573.986509:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628573.986530:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628573.986541:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628573/real 1778628573] req@ffff95434701d180 x1865013828657536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628573.986554:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828657536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628573.986562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628573.986566:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628573.986569:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628574.177873:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628574.177889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628574.945840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628575.201825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628575.201944:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628575.969819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628576.225874:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628576.225997:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628576.993842:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628577.249850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628577.249974:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628578.017767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628578.273799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628578.273907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628579.041893:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628579.105833:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628579.105847:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628579.105854:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628579.105869:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12913 00000100:00080000:2.0:1778628579.105887:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628579.105914:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628579.106045:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828657664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628579.106099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628579.106105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628579.106108:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628579.106116:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12913 00000100:00080000:2.0:1778628579.106124:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628579.106134:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628579.106153:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628579.106184:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828657792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628579.106191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828657664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628579.106272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628579.106277:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628579.106280:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628579.106286:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12913 00000100:00080000:2.0:1778628579.106294:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628579.106301:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778628579.106311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778628579.106320:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628579.106347:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828657920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628579.106364:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628579.106371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1778628579.106375:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628579/real 1778628579] req@ffff95435878b800 x1865013828657664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628579.106394:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628579.106405:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12913 00000100:00080000:2.0:1778628579.106420:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628579.106446:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628579.106456:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828657664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778628579.106461:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:1.0:1778628579.106507:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778628579.106516:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828658048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628579.106524:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778628579.106545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628579.106552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778628579.106557:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828657664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628579.106558:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628579.106585:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12913 00000100:00100000:1.0:1778628579.106631:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657792:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628579.106639:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628579.106652:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628579.106665:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:1.0:1778628579.106748:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628579.106765:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828658176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628579.106781:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628579/real 1778628579] req@ffff95435878a300 x1865013828657792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628579.106816:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828657792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628579.106836:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628579.106841:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628579.106846:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628579.106854:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628579.106879:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628579.106889:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628579/real 1778628579] req@ffff95435878ad80 x1865013828657920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628579.106916:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828657920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628579.106943:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628579.106947:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628579.106951:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828657920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628579.106959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628579.106985:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628579.106997:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628579/real 1778628579] req@ffff954358789180 x1865013828658048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628579.107010:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828658048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628579.107019:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628579.107034:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628579.107037:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628579.107060:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628579.107078:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628579.107088:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628579/real 1778628579] req@ffff954358788a80 x1865013828658176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628579.107100:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828658176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628579.107110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628579.107113:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628579.107124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628579.297796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628579.297899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628580.065839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628580.321835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628580.321931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628581.089841:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628581.345810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628581.345953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628582.113836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628582.369852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628582.370054:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628583.137892:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628583.393796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628583.393990:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628584.161822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628584.225854:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628584.225865:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628584.225870:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628584.225882:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12918 00000100:00080000:3.0:1778628584.225895:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628584.225904:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628584.225947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828658304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628584.225984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628584.225988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628584.225991:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628584.225995:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12918 00000100:00080000:3.0:1778628584.225999:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628584.226003:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628584.226013:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628584.226047:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828658432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628584.226060:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628584.226064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628584.226067:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628584.226072:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12918 00000100:00080000:3.0:1778628584.226076:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628584.226081:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628584.226087:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628584.226096:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828658304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628584.226101:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828658560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628584.226118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628584.226122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628584.226124:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628584.226128:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12918 00000100:00080000:3.0:1778628584.226132:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628584.226135:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628584.226141:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628584.226155:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828658688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628584.226165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628584.226169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628584.226171:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778628584.226184:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628584.226189:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12918 00000100:00080000:3.0:1778628584.226197:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628584.226203:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628584.226210:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628584.226225:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828658816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628584.226236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628584/real 1778628584] req@ffff95434701d880 x1865013828658304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628584.226264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828658304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628584.226277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628584.226283:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628584.226288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828658304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628584.226296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628584.226316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628584.226356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628584/real 1778628584] req@ffff95434701ea00 x1865013828658432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628584.226371:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828658432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628584.226381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628584.226386:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628584.226403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628584.226433:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628584.226458:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628584.226470:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628584/real 1778628584] req@ffff95434701c000 x1865013828658560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628584.226485:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828658560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628584.226526:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628584.226530:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628584.226535:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628584.226560:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628584.226582:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628584.226592:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628584/real 1778628584] req@ffff95434701c700 x1865013828658688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628584.226605:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828658688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628584.226613:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628584.226617:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628584.226620:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628584.226625:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628584.226639:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628584.226659:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628584/real 1778628584] req@ffff95434701f800 x1865013828658816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628584.226672:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828658816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628584.226681:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628584.226684:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628584.226729:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828658816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628584.417804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628584.417959:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628585.185808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628585.441837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628585.441965:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628586.209816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628586.465856:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628586.465963:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628587.233806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628587.489842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628587.489937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628588.257819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628588.513825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628588.513981:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628589.281790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628589.345904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628589.345918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628589.345925:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628589.345940:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12923 00000100:00080000:2.0:1778628589.345958:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628589.345968:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628589.346029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828658944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628589.346077:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628589.346082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628589.346084:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628589.346090:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12923 00000100:00080000:2.0:1778628589.346094:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628589.346127:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628589.346143:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628589.346167:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828659072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628589.346169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828658944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628589.346205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628589.346210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628589.346212:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628589.346218:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12923 00000100:00080000:2.0:1778628589.346223:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628589.346228:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628589.346234:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628589.346256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828659200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628589.346268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628589.346274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628589.346276:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628589.346281:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12923 00000100:00080000:2.0:1778628589.346285:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628589.346289:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628589.346295:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628589.346309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628589.346311:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828659328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628589.346327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628589.346339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628589.346342:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628589.346350:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12923 00000100:00080000:2.0:1778628589.346358:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628589.346365:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628589.346372:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628589.346377:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628589/real 1778628589] req@ffff954358789f80 x1865013828658944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628589.346413:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828659456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628589.346466:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828658944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628589.346488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628589.346495:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628589.346501:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828658944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628589.346537:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628589.346588:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628589.346617:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628589/real 1778628589] req@ffff95435878a680 x1865013828659072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628589.346651:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828659072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628589.346663:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628589.346667:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628589.346672:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628589.346681:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628589.346712:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628589.346793:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628589/real 1778628589] req@ffff954358788a80 x1865013828659200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628589.346820:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828659200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628589.346861:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628589.346865:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628589.346869:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628589.346877:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628589.346902:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628589.346914:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628589/real 1778628589] req@ffff954358789180 x1865013828659328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628589.346926:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828659328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628589.346942:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628589.346945:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628589.346949:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628589.346956:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628589.346976:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628589.346985:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628589/real 1778628589] req@ffff95435878ad80 x1865013828659456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628589.346997:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828659456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628589.347005:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628589.347008:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628589.347011:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628589.537783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628589.537862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628590.305753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628590.561813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628590.561928:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628591.329787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628591.585806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628591.585918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628592.353743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628592.609728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628592.609792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628593.377768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778628593.633877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778628593.633886:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628594.401831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628594.465893:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628594.465909:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628594.465916:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628594.465932:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12928 00000100:00080000:3.0:1778628594.465951:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628594.465962:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628594.466026:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828659584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628594.466072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628594.466077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628594.466103:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628594.466110:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12928 00000100:00080000:3.0:1778628594.466116:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628594.466121:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628594.466136:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628594.466153:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828659712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628594.466168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628594.466173:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628594.466176:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628594.466183:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12928 00000100:00080000:3.0:1778628594.466187:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628594.466191:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628594.466199:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628594.466213:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828659584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628594.466214:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828659840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628594.466240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628594.466244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628594.466247:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628594.466252:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12928 00000100:00080000:3.0:1778628594.466258:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628594.466262:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628594.466268:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628594.466326:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828659968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628594.466341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628594.466346:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628594.466349:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628594.466353:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12928 00000100:00080000:3.0:1778628594.466361:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628594.466368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628594.466376:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628594.466384:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628594.466402:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828660096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628594.466441:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628594/real 1778628594] req@ffff95434701d500 x1865013828659584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628594.466470:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828659584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628594.466499:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628594.466505:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628594.466511:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828659584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628594.466522:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628594.466555:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628594.466570:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628594/real 1778628594] req@ffff95434701c380 x1865013828659712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628594.466586:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828659712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628594.466597:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628594.466601:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628594.466605:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628594.466636:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628594.466656:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628594.466667:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628594/real 1778628594] req@ffff95434701f100 x1865013828659840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628594.466740:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828659840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628594.466751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628594.466755:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628594.466759:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628594.466777:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628594.466808:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628594.466822:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628594/real 1778628594] req@ffff95434701f800 x1865013828659968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628594.466836:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828659968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628594.466847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628594.466851:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628594.466855:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828659968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628594.466861:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628594.466878:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628594.466888:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628594/real 1778628594] req@ffff95434701c700 x1865013828660096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628594.466902:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828660096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628594.466912:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628594.466916:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628594.466919:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628594.657779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628594.657981:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628595.425912:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628595.681816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628595.682007:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628596.449862:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628596.705826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628596.705960:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628597.473762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628597.729787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628597.729876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628598.497744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628598.753871:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628598.754045:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628599.521757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628599.585788:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628599.585797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628599.585802:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628599.585811:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12934 00000100:00080000:3.0:1778628599.585823:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628599.585831:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628599.585870:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828660224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628599.585901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628599.585919:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628599.585922:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628599.585927:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12934 00000100:00080000:3.0:1778628599.585932:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628599.585938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628599.585949:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628599.585961:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828660224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628599.585966:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828660352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628599.585981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628599.585984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628599.585986:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628599.585991:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12934 00000100:00080000:3.0:1778628599.585994:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628599.585997:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628599.586002:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628599.586012:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828660480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628599.586021:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628599.586025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628599.586027:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628599.586030:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12934 00000100:00080000:3.0:1778628599.586034:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628599.586040:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628599.586043:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628599.586067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628599.586083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628599/real 1778628599] req@ffff95434701c000 x1865013828660224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628599.586084:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828660608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628599.586130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628599.586133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628599.586136:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628599.586141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828660224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628599.586145:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12934 00000100:00080000:3.0:1778628599.586154:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628599.586161:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:1.0:1778628599.586162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1778628599.586165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628599.586170:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628599.586176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828660224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628599.586198:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828660736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628599.586230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628599.586271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628599.586289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628599/real 1778628599] req@ffff95434701ea00 x1865013828660352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628599.586309:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828660352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628599.586323:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628599.586328:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628599.586333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628599.586343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628599.586370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628599.586381:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628599/real 1778628599] req@ffff95434701d880 x1865013828660480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628599.586412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828660480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628599.586424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628599.586428:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628599.586434:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628599.586444:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628599.586476:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628599.586490:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628599/real 1778628599] req@ffff95434701d180 x1865013828660608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628599.586509:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828660608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628599.586520:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628599.586526:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628599.586530:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628599.586540:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628599.586582:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628599.586596:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628599/real 1778628599] req@ffff95434701f480 x1865013828660736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628599.586608:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828660736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628599.586630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628599.586633:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628599.586637:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628599.777818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628599.777919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628600.545784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628600.802001:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628600.802277:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628601.569781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628601.826024:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628601.826202:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628602.593780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778628602.850651:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628602.850751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628603.617762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628603.873754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628603.873831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628604.641745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628604.705750:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628604.705758:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628604.705761:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628604.705769:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12939 00000100:00080000:3.0:1778628604.705778:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628604.705799:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628604.705832:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828660864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628604.705871:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628604.705888:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628604.705889:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628604.705894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12939 00000100:00080000:3.0:1778628604.705897:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628604.705900:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628604.705908:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628604.705913:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828660864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628604.705920:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828660992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628604.705929:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628604.705931:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628604.705932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628604.705935:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12939 00000100:00080000:3.0:1778628604.705937:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628604.705939:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628604.705942:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628604.705950:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828661120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628604.705956:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628604.705958:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628604.705959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628604.705961:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12939 00000100:00080000:3.0:1778628604.705963:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628604.705965:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628604.705975:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628604.705985:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628604.705987:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828661248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628604.705996:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628604.705998:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628604.706000:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628604.706003:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12939 00000100:00080000:3.0:1778628604.706006:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628604.706009:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628604.706011:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628604.706017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828661376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628604.706029:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628604/real 1778628604] req@ffff95434701ca80 x1865013828660864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628604.706051:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828660864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628604.706062:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628604.706067:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628604.706071:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828660864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628604.706110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628604.706133:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628604.706146:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628604/real 1778628604] req@ffff95434701e680 x1865013828660992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628604.706160:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828660992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628604.706185:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628604.706189:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628604.706204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828660992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628604.706214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628604.706238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628604.706249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628604/real 1778628604] req@ffff95434701df80 x1865013828661120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628604.706261:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828661120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628604.706269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628604.706272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628604.706275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628604.706282:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628604.706303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628604.706315:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628604/real 1778628604] req@ffff95434701f480 x1865013828661248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628604.706327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828661248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628604.706336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628604.706339:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628604.706351:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628604.706357:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628604.706380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628604.706391:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628604/real 1778628604] req@ffff95434701d180 x1865013828661376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628604.706401:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828661376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628604.706407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628604.706410:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628604.706412:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628604.897728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628604.897782:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628605.665750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628605.922272:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628605.922868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628606.689758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628606.945779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628606.945881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628607.713771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628607.969784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628607.969909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628608.737806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628608.993795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628608.993925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628609.761768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628609.825805:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628609.825816:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628609.825821:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628609.825833:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12944 00000100:00080000:2.0:1778628609.825847:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628609.825857:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628609.825902:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828661504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628609.825938:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628609.825943:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628609.825959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628609.825965:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12944 00000100:00080000:2.0:1778628609.825972:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628609.825978:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628609.826015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628609.826025:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828661504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628609.826040:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828661632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628609.826058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628609.826063:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628609.826066:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628609.826071:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12944 00000100:00080000:2.0:1778628609.826088:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628609.826093:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628609.826100:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778628609.826114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628609.826121:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828661760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628609.826136:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628609.826141:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628609.826143:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628609.826148:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12944 00000100:00080000:2.0:1778628609.826154:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628609.826162:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628609.826169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628609/real 1778628609] req@ffff95435878a300 x1865013828661504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778628609.826175:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628609.826197:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828661504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628609.826214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778628609.826216:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828661888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628609.826229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628609.826249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828661504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628609.826254:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628609.826267:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628609.826272:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628609.826280:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12944 00000100:00080000:2.0:1778628609.826287:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628609.826295:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628609.826304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661632:192.168.204.102@tcp:38:kworker.0 00000008:00100000:2.0:1778628609.826308:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628609.826338:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828662016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628609.826380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628609.826405:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628609/real 1778628609] req@ffff95435878b800 x1865013828661632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628609.826422:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828661632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628609.826433:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628609.826438:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628609.826442:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628609.826451:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628609.826499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628609.826519:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628609/real 1778628609] req@ffff954358789500 x1865013828661760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628609.826544:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828661760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628609.826562:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628609.826569:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628609.826576:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628609.826588:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628609.826619:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628609.826633:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628609/real 1778628609] req@ffff954358789880 x1865013828661888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628609.826668:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828661888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628609.826721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628609.826728:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628609.826734:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828661888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628609.826770:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628609.826811:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628609.826833:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628609/real 1778628609] req@ffff954358788700 x1865013828662016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628609.826858:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828662016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628609.826876:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628609.826882:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628609.826888:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628610.017794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628610.017934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628610.785770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628611.041782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628611.041904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628611.809773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628612.065769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628612.065904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628612.833833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628613.089787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628613.089937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628613.857778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628614.113757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628614.113891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628614.881733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628614.945826:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628614.945837:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628614.945843:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628614.945901:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12949 00000100:00080000:3.0:1778628614.945917:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628614.945962:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628614.946009:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828662144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628614.946056:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628614.946061:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628614.946063:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628614.946068:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12949 00000100:00080000:3.0:1778628614.946074:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628614.946078:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628614.946090:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628614.946132:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828662272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628614.946145:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628614.946150:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1778628614.946151:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828662144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628614.946152:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628614.946162:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12949 00000100:00080000:3.0:1778628614.946169:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628614.946172:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628614.946187:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628614.946212:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828662400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628614.946223:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628614.946227:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628614.946236:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628614.946241:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12949 00000100:00080000:3.0:1778628614.946246:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628614.946252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628614.946260:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628614.946268:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628614.946284:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828662528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628614.946296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628614/real 1778628614] req@ffff95434701d880 x1865013828662144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628614.946297:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628614.946310:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628614.946314:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628614.946344:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12949 00000100:00080000:3.0:1778628614.946351:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628614.946356:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628614.946362:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828662144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778628614.946367:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628614.946406:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828662656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628614.946466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628614.946475:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628614.946481:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828662144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628614.946538:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628614.946562:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628614.946574:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628614/real 1778628614] req@ffff95434701ea00 x1865013828662272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628614.946587:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828662272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628614.946595:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628614.946599:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628614.946602:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628614.946607:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628614.946620:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628614.946628:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628614/real 1778628614] req@ffff95434701c000 x1865013828662400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628614.946639:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828662400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628614.946646:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628614.946649:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628614.946651:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628614.946664:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628614.946684:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628614.946692:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628614/real 1778628614] req@ffff95434701c700 x1865013828662528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628614.946703:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828662528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628614.946711:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628614.946714:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628614.946716:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628614.946721:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628614.946735:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628614.946742:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628614/real 1778628614] req@ffff95434701f800 x1865013828662656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628614.946787:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828662656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628614.946799:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628614.946804:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628614.946808:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628615.137809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628615.137978:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628615.905761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628616.161797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628616.161934:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628616.929802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628617.185827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628617.185925:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628617.953780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628618.209796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628618.209910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628618.977793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628619.233836:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628619.233983:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628620.001807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628620.065775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628620.065804:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628620.065810:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628620.065822:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12954 00000100:00080000:3.0:1778628620.065835:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628620.065844:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628620.065888:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828662784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628620.065951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628620.065957:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628620.065959:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628620.065965:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12954 00000100:00080000:3.0:1778628620.065972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628620.065978:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628620.065991:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628620.066023:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828662912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628620.066029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828662784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628620.066060:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628620.066065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628620.066068:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628620.066075:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12954 00000100:00080000:3.0:1778628620.066079:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628620.066084:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628620.066090:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628620.066108:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828663040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628620.066121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628620.066126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628620.066128:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628620.066132:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12954 00000800:00000100:1.0:1778628620.066139:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628620.066155:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628620.066165:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628620.066174:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628620.066198:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828663168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628620.066205:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628620/real 1778628620] req@ffff95434701f100 x1865013828662784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628620.066254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628620.066259:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628620.066262:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628620.066268:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12954 00000100:00100000:1.0:1778628620.066273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828662784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628620.066284:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628620.066300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778628620.066307:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628620.066314:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628620.066316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628620.066332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828662784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628620.066344:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828663296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628620.066368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628620.066396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628620.066410:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628620/real 1778628620] req@ffff95434701c380 x1865013828662912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628620.066420:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828662912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628620.066428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628620.066431:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628620.066433:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828662912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628620.066439:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628620.066450:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628620.066456:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628620/real 1778628620] req@ffff95434701d500 x1865013828663040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628620.066476:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828663040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628620.066482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628620.066484:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628620.066486:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628620.066490:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628620.066505:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628620.066510:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628620/real 1778628620] req@ffff95434701f800 x1865013828663168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628620.066518:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828663168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628620.066524:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628620.066526:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628620.066527:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628620.066531:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628620.066542:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628620.066547:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628620/real 1778628620] req@ffff95434701c700 x1865013828663296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628620.066554:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828663296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628620.066569:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628620.066572:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628620.066574:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628620.257842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628620.257950:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628621.025837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628621.281830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628621.281968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628622.049823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628622.305817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628622.305958:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628623.073873:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628623.329797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628623.329966:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628624.097835:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628624.353790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628624.353916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628625.121796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628625.185837:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628625.185863:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628625.185869:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628625.185881:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12959 00000100:00080000:2.0:1778628625.185895:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628625.185906:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628625.185951:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828663424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628625.186001:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628625.186007:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628625.186009:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628625.186014:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12959 00000100:00080000:2.0:1778628625.186019:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628625.186023:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628625.186034:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628625.186062:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828663552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628625.186080:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628625.186087:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628625.186092:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628625.186096:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828663424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628625.186105:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12959 00000100:00080000:2.0:1778628625.186127:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628625.186134:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628625.186145:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628625.186173:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828663680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628625.186209:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778628625.186212:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628625.186222:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628625.186227:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628625.186238:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12959 00000100:00080000:2.0:1778628625.186246:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628625.186258:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628625.186267:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628625.186276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628625/real 1778628625] req@ffff954358788e00 x1865013828663424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628625.186290:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828663808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628625.186305:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628625.186309:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628625.186315:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628625.186317:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828663424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628625.186322:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12959 00000100:00080000:2.0:1778628625.186337:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628625.186347:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628625.186358:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628625.186377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778628625.186379:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828663936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628625.186396:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628625.186410:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828663424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628625.186471:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628625.186523:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628625.186555:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628625/real 1778628625] req@ffff954358789c00 x1865013828663552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628625.186581:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828663552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628625.186623:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628625.186631:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628625.186639:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628625.186652:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628625.186770:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628625.186788:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628625/real 1778628625] req@ffff95435878b100 x1865013828663680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628625.186813:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828663680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628625.186828:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628625.186835:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628625.186841:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628625.186854:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628625.186892:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628625.186907:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628625/real 1778628625] req@ffff95435878bb80 x1865013828663808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628625.186922:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828663808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628625.186932:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628625.186936:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628625.186951:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628625.186959:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628625.186995:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628625.187013:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628625/real 1778628625] req@ffff95435878b480 x1865013828663936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628625.187036:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828663936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628625.187062:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628625.187068:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628625.187074:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828663936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628625.377791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628625.377918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628626.145773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628626.401819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628626.401894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628627.169795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628627.425797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628627.425919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628628.193771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628628.449801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628628.449949:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628629.217772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628629.473859:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628629.473936:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628630.241799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628630.305843:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628630.305854:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628630.305859:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628630.305872:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12964 00000100:00080000:3.0:1778628630.305886:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628630.305896:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628630.305942:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828664064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628630.305978:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628630.306011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628630.306015:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628630.306021:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12964 00000100:00080000:3.0:1778628630.306028:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628630.306036:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628630.306048:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628630.306060:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828664064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628630.306071:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828664192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628630.306089:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628630.306094:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628630.306097:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628630.306103:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12964 00000100:00080000:3.0:1778628630.306108:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628630.306112:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628630.306118:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628630.306132:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828664320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628630.306145:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778628630.306153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628630.306156:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628630.306160:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628630.306168:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12964 00000100:00080000:3.0:1778628630.306174:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628630.306180:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628630.306197:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628630.306210:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628630/real 1778628630] req@ffff95434701c000 x1865013828664064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628630.306218:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828664448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628630.306232:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628630.306246:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628630.306251:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628630.306261:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12964 00000100:00080000:3.0:1778628630.306268:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628630.306277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828664064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628630.306277:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628630.306290:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628630.306309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778628630.306315:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828664576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628630.306331:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628630.306352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828664064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628630.306386:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628630.306420:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628630.306437:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628630/real 1778628630] req@ffff95434701ea00 x1865013828664192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628630.306453:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828664192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628630.306463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628630.306468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628630.306472:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628630.306479:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628630.306499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628630.306510:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628630/real 1778628630] req@ffff95434701d880 x1865013828664320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628630.306523:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828664320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628630.306533:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628630.306537:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628630.306540:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628630.306547:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628630.306565:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628630.306616:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628630/real 1778628630] req@ffff95434701d180 x1865013828664448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628630.306632:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828664448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628630.306643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628630.306647:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628630.306650:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628630.306657:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628630.306742:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628630.306765:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628630/real 1778628630] req@ffff95434701f480 x1865013828664576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628630.306790:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828664576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628630.306806:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628630.306812:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628630.306818:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628630.497794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628630.497889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628631.265760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628631.521786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628631.521852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628632.289774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628632.545753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628632.545824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628633.313792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628633.569732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628633.569796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628634.337743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628634.593761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628634.593815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628635.361796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628635.425770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628635.425779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628635.425791:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628635.425800:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12969 00000100:00080000:3.0:1778628635.425813:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628635.425821:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628635.425871:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828664704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628635.425899:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628635.425903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628635.425904:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628635.425908:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12969 00000100:00080000:3.0:1778628635.425935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628635.425939:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628635.425948:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628635.425960:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828664832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628635.425971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628635.425974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628635.425976:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628635.425980:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12969 00000100:00080000:3.0:1778628635.425983:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628635.425986:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628635.425993:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828664704:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778628635.425993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628635.426008:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828664960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628635.426017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628635.426021:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628635.426022:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628635.426026:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12969 00000100:00080000:3.0:1778628635.426029:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628635.426033:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628635.426037:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628635.426048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828665088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628635.426082:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628635.426085:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628635.426087:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628635.426090:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12969 00000100:00080000:3.0:1778628635.426096:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628635.426099:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628635.426104:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628635.426120:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778628635.426126:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828665216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628635.426187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628635.426208:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628635.426237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628635.617738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628635.617835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628636.385744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628636.641749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628636.641898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628637.409743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628637.665818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628637.666042:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628638.433836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778628638.498107:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778628638.498174:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778628638.498218:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628638.498284:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628638.498295:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628638.498305:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628638.498314:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778628638.498531:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628635/real 1778628638] req@ffff95434701ca80 x1865013828664960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628638.498580:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828664960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628638.498602:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628638.498613:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628638.498621:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628638.498637:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628635/real 1778628638] req@ffff95434701f480 x1865013828665088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628638.498660:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828665088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628638.498751:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628638.498761:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628638.498769:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628638.498785:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628635/real 1778628638] req@ffff95434701d180 x1865013828665216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628638.498842:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828665216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628638.498859:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628638.498866:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628638.498875:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628638.498893:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628635/real 1778628638] req@ffff95434701df80 x1865013828664704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628638.498918:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828664704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628638.498933:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628638.498939:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628638.498946:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828664704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628638.498958:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628635/real 1778628638] req@ffff95434701e680 x1865013828664832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628638.498981:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828664832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628638.498997:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628638.499004:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628638.499010:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828664832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628638.689799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628638.689952:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628639.457764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628639.713852:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628639.713980:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628640.481820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628640.545786:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628640.545796:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628640.545799:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628640.545809:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12975 00000100:00080000:3.0:1778628640.545819:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628640.545827:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628640.545867:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828665344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628640.545903:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628640.545907:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628640.545908:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628640.545912:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12975 00000100:00080000:3.0:1778628640.545915:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628640.545918:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628640.545943:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628640.545958:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828665472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628640.545967:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628640.545970:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628640.545972:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628640.545975:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12975 00000100:00080000:3.0:1778628640.545978:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628640.545982:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628640.545986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628640.546001:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828665600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628640.546008:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828665344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628640.546012:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628640.546020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628640.546022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628640.546033:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12975 00000100:00080000:3.0:1778628640.546036:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628640.546039:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628640.546044:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628640.546060:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828665728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628640.546070:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628640.546074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628640.546085:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628640.546090:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12975 00000100:00080000:3.0:1778628640.546096:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628640.546102:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628640.546109:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628640.546128:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828665856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628640.546170:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628640.546242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628640/real 1778628640] req@ffff95434701d880 x1865013828665344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628640.546284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828665344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628640.546309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628640.546320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628640.546328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828665344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628640.546346:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628640.546431:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628640.546459:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628640/real 1778628640] req@ffff95434701ea00 x1865013828665472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628640.546486:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828665472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628640.546503:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628640.546510:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628640.546518:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628640.546570:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628640.546615:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628640.546638:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628640/real 1778628640] req@ffff95434701c000 x1865013828665600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628640.546766:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828665600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628640.546783:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628640.546790:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628640.546796:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628640.546812:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628640.546861:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628640.546885:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628640/real 1778628640] req@ffff95434701c700 x1865013828665728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628640.546910:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828665728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628640.546937:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628640.546946:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628640.546951:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628640.546964:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628640.547006:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628640.547027:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628640/real 1778628640] req@ffff95434701f800 x1865013828665856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628640.547049:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828665856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628640.547093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628640.547100:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628640.547106:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828665856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628640.737788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628640.737927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628641.505762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628641.761797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628641.761906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628642.529758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628642.785760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628642.785861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628643.553821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628643.809748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628643.809859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628644.577757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628644.833877:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628644.833899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628645.601780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628645.665775:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628645.665783:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628645.665787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628645.665795:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12980 00000100:00080000:3.0:1778628645.665804:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628645.665835:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628645.665868:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828665984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628645.665891:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628645.665893:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628645.665895:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628645.665898:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12980 00000100:00080000:3.0:1778628645.665900:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628645.665902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628645.665910:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628645.665917:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828666112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628645.665942:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628645.665945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628645.665946:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628645.665949:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12980 00000100:00080000:3.0:1778628645.665952:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628645.665955:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628645.665958:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628645.665967:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828665984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628645.665968:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828666240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628645.665980:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628645.665982:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628645.665983:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628645.665986:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12980 00000100:00080000:3.0:1778628645.665989:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628645.665991:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628645.665995:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628645.666004:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828666368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628645.666010:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628645.666012:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628645.666013:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628645.666015:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12980 00000100:00080000:3.0:1778628645.666022:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628645.666025:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778628645.666029:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778628645.666030:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628645.666040:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828666496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628645.666062:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628645/real 1778628645] req@ffff95434701d500 x1865013828665984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628645.666078:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828665984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628645.666085:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628645.666089:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628645.666092:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828665984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628645.666097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628645.666110:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628645.666117:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628645/real 1778628645] req@ffff95434701c380 x1865013828666112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628645.666125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828666112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628645.666138:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628645.666140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628645.666142:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628645.666160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628645.666172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628645.666178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628645/real 1778628645] req@ffff95434701f100 x1865013828666240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628645.666185:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828666240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628645.666190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628645.666192:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628645.666194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628645.666197:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628645.666207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628645.666212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628645/real 1778628645] req@ffff95434701f800 x1865013828666368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628645.666218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828666368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628645.666234:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628645.666236:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628645.666243:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628645.666247:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628645.666261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628645.666267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628645/real 1778628645] req@ffff95434701c700 x1865013828666496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628645.666293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828666496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628645.666299:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628645.666301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628645.666303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628645.857772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628645.857865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628646.625735:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628646.881770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628646.881774:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628647.649773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628647.905749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628647.905814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628648.673720:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628648.929740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628648.929831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628649.697749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628649.953798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628649.953881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628650.721741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628650.785783:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628650.785794:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628650.785799:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628650.785828:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12985 00000100:00080000:2.0:1778628650.785841:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628650.785849:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628650.785887:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828666624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628650.785919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628650.785927:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628650.785929:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628650.785934:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12985 00000100:00080000:2.0:1778628650.785938:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628650.785944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628650.785954:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628650.785964:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828666624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628650.785971:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828666752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628650.785985:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628650.785988:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628650.785990:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628650.785994:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12985 00000100:00080000:2.0:1778628650.785999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628650.786002:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628650.786007:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628650.786020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828666880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628650.786024:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628650.786049:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628650.786053:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628650.786055:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628650.786057:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628650/real 1778628650] req@ffff954358788380 x1865013828666624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628650.786064:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12985 00000100:00080000:2.0:1778628650.786073:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628650.786078:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628650.786081:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828666624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778628650.786087:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:1.0:1778628650.786092:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628650.786096:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628650.786104:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828667008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628650.786109:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828666624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628650.786124:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628650.786128:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628650.786129:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628650.786135:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12985 00000100:00100000:1.0:1778628650.786137:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666752:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628650.786142:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628650.786148:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628650.786154:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:1.0:1778628650.786160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628650.786169:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828667136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628650.786188:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628650/real 1778628650] req@ffff95435878aa00 x1865013828666752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628650.786215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828666752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628650.786222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628650.786225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628650.786227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628650.786233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628650.786247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628650.786256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628650/real 1778628650] req@ffff95435878b480 x1865013828666880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628650.786267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828666880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628650.786275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628650.786279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628650.786282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828666880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628650.786288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628650.786309:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628650.786326:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628650/real 1778628650] req@ffff95435878bb80 x1865013828667008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628650.786334:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828667008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628650.786340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628650.786343:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628650.786346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628650.786364:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628650.786384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628650.786395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628650/real 1778628650] req@ffff95435878b100 x1865013828667136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628650.786406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828667136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628650.786412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628650.786415:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628650.786417:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628650.977737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628650.977799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628651.745740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628652.001757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628652.001805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628652.769777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628653.025762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628653.025872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628653.793753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628654.049769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628654.049995:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628654.817752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628655.073757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628655.073815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628655.841748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628655.905777:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628655.905786:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628655.905791:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628655.905800:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12990 00000100:00080000:3.0:1778628655.905811:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628655.905818:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628655.905852:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828667264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628655.905879:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628655.905883:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628655.905885:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628655.905889:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12990 00000100:00080000:3.0:1778628655.905893:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628655.905896:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628655.905905:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628655.905917:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828667392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628655.905927:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828667264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628655.905932:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628655.905949:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628655.905951:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628655.905956:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12990 00000100:00080000:3.0:1778628655.905961:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628655.905965:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628655.905970:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628655.905982:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828667520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628655.905993:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778628655.905998:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628655.905999:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628655.906001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628655.906006:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12990 00000100:00080000:3.0:1778628655.906013:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628655.906017:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628655.906022:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628655.906033:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828667648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628655.906044:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628655.906048:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628655/real 1778628655] req@ffff95434701c000 x1865013828667264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628655.906049:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628655.906053:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628655.906062:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12990 00000100:00080000:3.0:1778628655.906067:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628655.906083:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628655.906086:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828667264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778628655.906098:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628655.906100:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628655.906106:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628655.906109:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828667264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628655.906116:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828667776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628655.906136:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628655.906158:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628655.906167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628655/real 1778628655] req@ffff95434701ea00 x1865013828667392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628655.906176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828667392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628655.906183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628655.906185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628655.906187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628655.906191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628655.906202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628655.906207:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628655/real 1778628655] req@ffff95434701d880 x1865013828667520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628655.906215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828667520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628655.906229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628655.906232:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628655.906234:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628655.906238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628655.906251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628655.906257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628655/real 1778628655] req@ffff95434701e680 x1865013828667648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628655.906264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828667648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628655.906269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628655.906271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628655.906273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628655.906283:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628655.906293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628655.906298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628655/real 1778628655] req@ffff95434701df80 x1865013828667776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628655.906306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828667776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628655.906311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628655.906313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628655.906320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828667776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628656.097757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628656.097829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628656.865869:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628657.121799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628657.121857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628657.889751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628658.145750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628658.145826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628658.913773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628659.169766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628659.169867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628659.937749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628660.193742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628660.193874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628660.961830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628661.025787:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628661.025799:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628661.025804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628661.025814:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 12995 00000100:00080000:3.0:1778628661.025836:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628661.025843:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628661.025888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828667904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628661.025915:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628661.025918:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628661.025920:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628661.025923:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12995 00000100:00080000:3.0:1778628661.025926:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628661.025928:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628661.025937:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628661.025947:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828668032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628661.025954:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628661.025957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628661.025958:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628661.025960:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12995 00000100:00100000:1.0:1778628661.025992:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828667904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628661.025995:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628661.026000:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628661.026004:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628661.026016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828668160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628661.026023:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628661.026026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628661.026027:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628661.026031:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12995 00000100:00080000:3.0:1778628661.026033:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628661.026035:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628661.026039:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628661.026052:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828668288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628661.026062:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628661.026066:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628661.026067:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628661.026071:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 12995 00000100:00080000:3.0:1778628661.026076:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628661.026080:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628661.026086:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:1.0:1778628661.026093:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628661.026099:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828668416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628661.026141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628661/real 1778628661] req@ffff95434701d180 x1865013828667904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628661.026159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828667904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628661.026169:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628661.026173:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628661.026176:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828667904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628661.026183:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628661.026202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628661.026211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628661/real 1778628661] req@ffff95434701f480 x1865013828668032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628661.026220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828668032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628661.026226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628661.026228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628661.026231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628661.026253:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628661.026275:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628661.026282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628661/real 1778628661] req@ffff95434701ca80 x1865013828668160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628661.026290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828668160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628661.026296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628661.026299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628661.026301:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628661.026305:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628661.026315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628661.026320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628661/real 1778628661] req@ffff95434701df80 x1865013828668288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628661.026328:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828668288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628661.026334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628661.026335:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628661.026337:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628661.026341:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628661.026350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628661.026356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628661/real 1778628661] req@ffff95434701e680 x1865013828668416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628661.026373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828668416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628661.026380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628661.026382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628661.026384:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628661.217761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628661.217813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628661.985749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628662.241737:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628662.241779:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628663.009771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628663.265741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628663.265786:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628664.033797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628664.289778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628664.289873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628665.057761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628665.313804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628665.313844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628666.081754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628666.145772:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628666.145781:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628666.145785:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628666.145793:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13000 00000100:00080000:3.0:1778628666.145803:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628666.145809:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628666.145850:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828668544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628666.145885:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628666.145888:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628666.145890:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628666.145893:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13000 00000100:00080000:3.0:1778628666.145896:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628666.145898:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628666.145906:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628666.145916:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828668672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628666.145923:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628666.145925:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628666.145926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628666.145928:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13000 00000100:00080000:3.0:1778628666.145930:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628666.145932:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628666.145972:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628666.145984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828668800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628666.145985:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828668544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628666.146010:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628666.146013:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628666.146014:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628666.146019:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13000 00000100:00080000:3.0:1778628666.146022:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628666.146025:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628666.146029:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628666.146037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828668928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628666.146043:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628666.146046:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628666.146048:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628666.146051:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13000 00000100:00080000:3.0:1778628666.146053:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628666.146055:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628666.146057:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628666.146065:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828669056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628666.146081:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628666.146129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628666/real 1778628666] req@ffff95434701d880 x1865013828668544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628666.146147:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828668544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628666.146156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628666.146160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628666.146163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828668544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628666.146169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628666.146202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628666.146210:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628666/real 1778628666] req@ffff95434701ea00 x1865013828668672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628666.146218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828668672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628666.146224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628666.146227:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628666.146229:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628666.146251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628666.146261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628666.146266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628666/real 1778628666] req@ffff95434701c000 x1865013828668800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628666.146274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828668800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628666.146279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628666.146281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628666.146283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628666.146287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628666.146297:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628666.146302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628666/real 1778628666] req@ffff95434701c700 x1865013828668928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628666.146319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828668928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628666.146325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628666.146327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628666.146329:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828668928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628666.146333:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628666.146346:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628666.146351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628666/real 1778628666] req@ffff95434701f800 x1865013828669056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628666.146358:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828669056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628666.146364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628666.146366:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628666.146368:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628666.337745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628666.337861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628667.105759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628667.362233:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628667.362454:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628668.129810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628668.385779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628668.386175:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628669.153764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628669.409898:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628669.410447:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628670.177776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628670.433765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628670.433941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628671.201820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628671.265831:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628671.265857:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628671.265861:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628671.265870:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13005 00000100:00080000:3.0:1778628671.265882:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628671.265889:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628671.265925:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828669184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628671.265953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628671.265956:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628671.265958:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628671.265962:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13005 00000100:00080000:3.0:1778628671.265965:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628671.265968:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628671.266002:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628671.266014:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828669312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628671.266025:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628671.266040:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628671.266043:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628671.266066:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828669184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628671.266069:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13005 00000100:00080000:3.0:1778628671.266079:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628671.266085:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628671.266109:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628671.266128:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828669440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628671.266144:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628671.266147:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1778628671.266154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628671.266160:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628671.266167:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13005 00000100:00080000:3.0:1778628671.266173:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628671.266178:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628671.266184:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628671.266202:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828669568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628671.266211:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628671.266214:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628671.266216:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628671.266217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628671/real 1778628671] req@ffff95434701f100 x1865013828669184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628671.266221:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13005 00000100:00080000:3.0:1778628671.266227:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628671.266235:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628671.266245:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628671.266256:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828669696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628671.266268:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828669184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628671.266291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628671.266298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628671.266305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828669184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628671.266316:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628671.266352:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628671.266385:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628671/real 1778628671] req@ffff95434701c380 x1865013828669312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628671.266399:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828669312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628671.266406:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628671.266409:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628671.266412:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628671.266451:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628671.266471:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628671.266480:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628671/real 1778628671] req@ffff95434701d500 x1865013828669440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628671.266503:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828669440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628671.266510:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628671.266513:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628671.266516:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628671.266529:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628671.266552:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628671.266588:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628671/real 1778628671] req@ffff95434701f800 x1865013828669568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628671.266608:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828669568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628671.266622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628671.266627:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628671.266632:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628671.266642:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628671.266726:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628671.266739:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628671/real 1778628671] req@ffff95434701c700 x1865013828669696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628671.266750:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828669696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628671.266790:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628671.266793:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628671.266796:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669696:192.168.204.102@tcp:8:kworker.0 00010000:00010000:2.0:1778628671.458173:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628671.458249:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628672.225816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628672.481764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628672.481833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628673.249787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628673.505827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628673.505888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628674.273785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628674.529768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628674.529890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628675.297834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628675.553749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628675.553858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628676.321758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628676.385806:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628676.385817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628676.385822:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628676.385833:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13010 00000100:00080000:1.0:1778628676.385855:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628676.385862:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628676.385895:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828669824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628676.385926:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628676.385930:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628676.385932:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628676.385935:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13010 00000100:00080000:1.0:1778628676.385938:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628676.385942:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628676.385951:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628676.385963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828669952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628676.385983:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828669824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628676.385985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628676.385992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628676.385996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628676.386001:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13010 00000100:00080000:1.0:1778628676.386003:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628676.386006:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628676.386010:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628676.386048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828670080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628676.386057:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628676.386060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628676.386062:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628676.386068:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13010 00000800:00000100:0.0:1778628676.386070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628676.386074:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628676.386080:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628676.386086:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628676.386097:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828670208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628676.386105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628676.386109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628676.386111:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628676.386117:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13010 00000100:00100000:0.0:1778628676.386119:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628676/real 1778628676] req@ffff954350197100 x1865013828669824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628676.386120:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628676.386126:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628676.386134:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628676.386146:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828670336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628676.386163:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828669824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628676.386175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628676.386180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628676.386185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828669824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628676.386207:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628676.386240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628676.386255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628676/real 1778628676] req@ffff954350194000 x1865013828669952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628676.386273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828669952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628676.386284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628676.386289:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628676.386293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828669952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628676.386302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628676.386329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628676.386340:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628676/real 1778628676] req@ffff954350196d80 x1865013828670080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628676.386350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828670080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628676.386358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628676.386361:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628676.386363:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628676.386368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628676.386384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628676.386391:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628676/real 1778628676] req@ffff954350197480 x1865013828670208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628676.386414:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828670208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628676.386421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628676.386424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628676.386427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628676.386432:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628676.386447:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628676.386454:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628676/real 1778628676] req@ffff954350197800 x1865013828670336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628676.386464:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828670336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628676.386471:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628676.386473:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628676.386475:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628676.577744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628676.577811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628677.345785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628677.601751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628677.601896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628678.369819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628678.625776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628678.625878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628679.393801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628679.649763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628679.649847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628680.417727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628680.673756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628680.673826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628681.441742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628681.505743:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628681.505751:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628681.505755:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628681.505774:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13016 00000100:00080000:0.0:1778628681.505783:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628681.505788:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628681.505818:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828670464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628681.505841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628681.505844:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628681.505846:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628681.505849:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13016 00000100:00080000:0.0:1778628681.505852:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628681.505856:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628681.505863:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628681.505875:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828670592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628681.505876:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828670464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628681.505890:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628681.505892:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628681.505893:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628681.505895:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13016 00000100:00080000:0.0:1778628681.505897:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628681.505900:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628681.505903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628681.505912:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828670720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628681.505928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628681.505931:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628681.505932:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628681.505935:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13016 00000100:00080000:0.0:1778628681.505939:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628681.505944:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628681.505946:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628681.505951:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628681.505963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828670848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628681.505984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628681.505988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628681.505992:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628681.505992:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628681/real 1778628681] req@ffff954342f4b100 x1865013828670464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628681.506003:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13016 00000100:00080000:0.0:1778628681.506009:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628681.506013:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628681.506017:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828670464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778628681.506021:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628681.506031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628681.506035:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628681.506038:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828670976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628681.506042:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828670464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628681.506067:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628681.506086:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628681.506094:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628681/real 1778628681] req@ffff954342f48700 x1865013828670592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628681.506103:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828670592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628681.506108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628681.506110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628681.506113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628681.506116:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628681.506126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628681.506131:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628681/real 1778628681] req@ffff954342f49c00 x1865013828670720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628681.506146:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828670720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628681.506152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628681.506154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628681.506156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628681.506159:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628681.506172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628681.506177:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628681/real 1778628681] req@ffff954342f4b480 x1865013828670848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628681.506185:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828670848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628681.506190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628681.506191:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628681.506193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628681.506196:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628681.506206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628681.506211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628681/real 1778628681] req@ffff954342f4a300 x1865013828670976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628681.506218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828670976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628681.506233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628681.506235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628681.506238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828670976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628681.697740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628681.697815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628682.465733:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628682.721744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628682.721827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628683.489713:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628683.745730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628683.745799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628684.513782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628684.769751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628684.769824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628685.537788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628685.793747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628685.793849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628686.561780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628686.625818:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628686.625829:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628686.625834:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628686.625846:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13021 00000100:00080000:0.0:1778628686.625859:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628686.625869:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628686.625911:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828671104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628686.625949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628686.625953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628686.625956:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628686.625961:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13021 00000100:00080000:0.0:1778628686.625965:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628686.625969:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628686.625979:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628686.625993:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828671232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628686.626005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628686.626010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628686.626013:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628686.626017:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13021 00000100:00080000:0.0:1778628686.626020:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628686.626024:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628686.626029:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628686.626061:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828671104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628686.626070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828671360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628686.626084:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628686.626088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628686.626091:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628686.626095:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13021 00000100:00080000:0.0:1778628686.626100:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628686.626105:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628686.626111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628686.626134:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828671488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628686.626145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628686.626149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628686.626151:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628686.626155:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13021 00000100:00080000:0.0:1778628686.626159:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628686.626168:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628686.626169:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628686.626178:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628686.626192:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828671616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628686.626231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628686/real 1778628686] req@ffff954342f49880 x1865013828671104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628686.626264:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828671104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628686.626284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628686.626292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628686.626299:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828671104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628686.626313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628686.626343:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628686.626367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628686/real 1778628686] req@ffff954342f4bb80 x1865013828671232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628686.626389:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828671232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628686.626442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628686.626448:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628686.626453:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628686.626520:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628686.626549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628686.626567:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628686/real 1778628686] req@ffff954342f48e00 x1865013828671360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628686.626585:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828671360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628686.626597:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628686.626602:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628686.626607:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628686.626615:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628686.626636:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628686.626648:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628686/real 1778628686] req@ffff954342f4aa00 x1865013828671488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628686.626665:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828671488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628686.626715:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628686.626719:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628686.626760:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628686.626771:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628686.626805:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628686.626827:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628686/real 1778628686] req@ffff954342f49180 x1865013828671616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628686.626849:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828671616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628686.626864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628686.626870:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628686.626876:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628686.817791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628686.817890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628687.585795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628687.841744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628687.841830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628688.609740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628688.865776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628688.865827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628689.633755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628689.889724:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628689.889805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628690.657788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628690.913789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628690.913864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628691.681777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628691.745823:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628691.745834:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628691.745857:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628691.745868:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13026 00000100:00080000:3.0:1778628691.745881:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628691.745889:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628691.745930:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828671744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628691.745973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628691.745976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628691.745978:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628691.745982:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13026 00000100:00080000:3.0:1778628691.745984:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628691.745987:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628691.745996:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628691.746005:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828671872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628691.746011:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628691.746014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628691.746015:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628691.746017:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13026 00000100:00080000:3.0:1778628691.746019:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628691.746021:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628691.746024:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628691.746030:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828672000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628691.746050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628691.746053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628691.746054:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628691.746058:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13026 00000100:00080000:3.0:1778628691.746061:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628691.746064:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628691.746068:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628691.746072:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828671744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628691.746079:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828672128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628691.746087:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628691.746089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628691.746090:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628691.746093:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13026 00000100:00080000:3.0:1778628691.746095:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628691.746098:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628691.746102:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628691.746111:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828672256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628691.746149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628691.746185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628691/real 1778628691] req@ffff95434701c000 x1865013828671744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628691.746202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828671744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628691.746211:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628691.746215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628691.746224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828671744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628691.746230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628691.746247:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628691.746255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628691/real 1778628691] req@ffff95434701ea00 x1865013828671872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628691.746263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828671872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628691.746269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628691.746271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628691.746273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828671872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628691.746277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628691.746287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628691.746292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628691/real 1778628691] req@ffff95434701d880 x1865013828672000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628691.746298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828672000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628691.746304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628691.746306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628691.746307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628691.746330:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628691.746341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628691.746353:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628691/real 1778628691] req@ffff95434701e680 x1865013828672128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628691.746379:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828672128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628691.746386:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628691.746388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628691.746389:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628691.746394:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628691.746419:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628691.746426:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628691/real 1778628691] req@ffff95434701df80 x1865013828672256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628691.746433:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828672256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628691.746440:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628691.746442:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628691.746444:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628691.937750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628691.937877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628692.705748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628692.961767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628692.961861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628693.729727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628693.985759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628693.985853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628694.753769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628695.009777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628695.009825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628695.777854:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628696.033836:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628696.033938:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628696.801805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628696.865801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628696.865815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628696.865822:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628696.865835:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13031 00000100:00080000:3.0:1778628696.865879:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628696.865888:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628696.865935:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828672384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628696.865973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628696.865979:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628696.865983:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628696.865991:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13031 00000100:00080000:3.0:1778628696.865998:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628696.866004:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628696.866019:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628696.866033:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828672384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628696.866044:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828672512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628696.866059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628696.866063:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628696.866065:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628696.866069:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13031 00000100:00080000:3.0:1778628696.866073:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628696.866077:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628696.866083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628696.866096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828672640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628696.866105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628696.866116:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628696.866119:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628696.866123:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13031 00000100:00080000:3.0:1778628696.866129:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628696.866132:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628696.866137:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628696.866143:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628696.866161:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828672768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628696.866172:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628696.866176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628696.866178:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628696.866181:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13031 00000100:00080000:3.0:1778628696.866186:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628696.866190:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628696.866194:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628696.866202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628696/real 1778628696] req@ffff95434701ca80 x1865013828672384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628696.866207:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828672896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628696.866281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828672384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628696.866296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628696.866301:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628696.866307:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828672384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628696.866335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628696.866359:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628696.866371:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628696/real 1778628696] req@ffff95434701f480 x1865013828672512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628696.866383:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828672512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628696.866392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628696.866395:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628696.866398:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628696.866403:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628696.866417:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628696.866424:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628696/real 1778628696] req@ffff95434701d180 x1865013828672640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628696.866435:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828672640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628696.866456:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628696.866460:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628696.866463:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628696.866469:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628696.866489:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628696.866497:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628696/real 1778628696] req@ffff95434701df80 x1865013828672768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628696.866508:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828672768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628696.866516:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628696.866518:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628696.866521:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628696.866527:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628696.866540:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628696.866548:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628696/real 1778628696] req@ffff95434701e680 x1865013828672896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628696.866558:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828672896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628696.866566:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628696.866569:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628696.866577:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828672896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628697.057792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628697.057890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628697.825796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628698.081841:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628698.081985:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628698.849761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628699.105849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628699.105933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628699.873815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628700.129781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628700.129879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628700.897756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628701.153784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628701.153916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628701.921787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628701.985727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628701.985734:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628701.985739:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628701.985749:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13036 00000100:00080000:3.0:1778628701.985759:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628701.985767:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628701.985804:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828673024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628701.985842:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628701.985845:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628701.985846:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628701.985849:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13036 00000100:00080000:3.0:1778628701.985853:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628701.985856:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628701.985863:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628701.985874:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828673152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628701.985882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628701.985885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628701.985886:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628701.985891:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13036 00000100:00100000:1.0:1778628701.985892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828673024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628701.985893:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628701.985899:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628701.985903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628701.985910:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828673280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628701.985917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628701.985919:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628701.985920:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628701.985923:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13036 00000100:00080000:3.0:1778628701.985930:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628701.985934:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628701.985937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628701.985945:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828673408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628701.985952:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628701.985954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628701.985955:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628701.985957:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13036 00000100:00080000:3.0:1778628701.985960:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628701.985962:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628701.985964:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628701.985972:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828673536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628701.985992:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628701.986012:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628701.986023:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628701.986037:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628702.177782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628702.177851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628702.945816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628703.201792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628703.201907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628703.969797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628704.225808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628704.225871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628704.993819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778628705.057885:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778628705.057922:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778628705.057937:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628705.057976:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628705.057984:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628705.057993:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628705.058003:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778628705.058113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628701/real 1778628705] req@ffff95434701ea00 x1865013828673152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628705.058158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828673152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628705.058172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628705.058178:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628705.058183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628705.058191:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628701/real 1778628705] req@ffff95434701c000 x1865013828673280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628705.058237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828673280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628705.058245:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628705.058250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628705.058253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628705.058260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628701/real 1778628705] req@ffff95434701c700 x1865013828673408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628705.058271:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828673408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628705.058279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628705.058282:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628705.058285:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628705.058293:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628701/real 1778628705] req@ffff95434701f800 x1865013828673536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628705.058306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828673536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628705.058314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628705.058318:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628705.058321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628705.058328:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628701/real 1778628705] req@ffff95434701d880 x1865013828673024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628705.058349:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828673024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628705.058358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628705.058362:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628705.058366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828673024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628705.249730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628705.249937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628706.017815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628706.273755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628706.273883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628707.041778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628707.105824:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628707.105835:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628707.105839:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628707.105849:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13041 00000100:00080000:1.0:1778628707.105862:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628707.105868:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628707.105910:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828673664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628707.105943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628707.105946:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628707.105948:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628707.105951:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13041 00000100:00080000:1.0:1778628707.105954:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628707.105956:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628707.105964:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628707.105974:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828673792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628707.106007:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828673664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628707.106023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628707.106026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628707.106028:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628707.106032:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13041 00000100:00080000:1.0:1778628707.106035:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628707.106038:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628707.106043:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628707.106052:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828673920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628707.106076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628707.106078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628707.106080:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628707.106083:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13041 00000100:00080000:1.0:1778628707.106097:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628707.106100:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628707.106104:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628707.106118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628707.106121:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828674048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628707.106130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628707.106133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628707.106135:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628707.106139:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13041 00000100:00080000:1.0:1778628707.106150:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628707.106154:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628707.106159:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628707.106165:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628707/real 1778628707] req@ffff954350194700 x1865013828673664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628707.106170:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828674176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628707.106203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828673664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628707.106214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628707.106218:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628707.106221:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828673664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628707.106245:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628707.106286:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628707.106304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628707/real 1778628707] req@ffff954350194380 x1865013828673792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628707.106319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828673792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628707.106329:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628707.106333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628707.106336:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628707.106342:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628707.106363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628707.106373:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628707/real 1778628707] req@ffff954350194a80 x1865013828673920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628707.106386:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828673920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628707.106395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628707.106399:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628707.106403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828673920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628707.106409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628707.106437:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628707.106449:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628707/real 1778628707] req@ffff954350196a00 x1865013828674048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628707.106478:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828674048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628707.106488:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628707.106491:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628707.106495:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628707.106501:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628707.106526:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628707.106537:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628707/real 1778628707] req@ffff954350194e00 x1865013828674176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628707.106548:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828674176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628707.106558:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628707.106562:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628707.106565:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628707.297760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628707.297867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628708.065766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628708.321795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628708.321895:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628709.089766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628709.345815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628709.345893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628710.113807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628710.369822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628710.369866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628711.137767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628711.393778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628711.393816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628712.161826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628712.225841:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628712.225853:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628712.225858:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628712.225869:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13046 00000100:00080000:0.0:1778628712.225883:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628712.225889:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628712.225932:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828674304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628712.225975:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628712.225979:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628712.225980:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628712.225984:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13046 00000100:00080000:0.0:1778628712.225987:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628712.225990:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628712.225999:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628712.226010:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828674432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628712.226019:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628712.226021:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628712.226023:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628712.226029:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13046 00000100:00100000:1.0:1778628712.226029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828674304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628712.226039:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628712.226044:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628712.226068:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628712.226080:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828674560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628712.226087:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628712.226090:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628712.226092:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628712.226095:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13046 00000100:00080000:0.0:1778628712.226106:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628712.226109:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628712.226113:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628712.226122:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828674688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628712.226124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628712.226132:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628712.226134:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628712.226136:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628712.226139:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13046 00000100:00080000:0.0:1778628712.226141:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628712.226143:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628712.226146:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628712.226153:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828674816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628712.226179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628712/real 1778628712] req@ffff954342f48380 x1865013828674304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628712.226204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828674304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628712.226217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628712.226222:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628712.226228:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828674304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628712.226259:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628712.226287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628712.226320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628712/real 1778628712] req@ffff954342f48000 x1865013828674432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628712.226340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828674432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628712.226353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628712.226359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628712.226375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628712.226385:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628712.226418:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628712.226434:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628712/real 1778628712] req@ffff954342f49180 x1865013828674560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628712.226452:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828674560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628712.226464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628712.226468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628712.226473:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628712.226482:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628712.226511:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628712.226526:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628712/real 1778628712] req@ffff954342f4aa00 x1865013828674688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628712.226544:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828674688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628712.226555:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628712.226559:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628712.226564:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628712.226573:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628712.226601:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628712.226628:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628712/real 1778628712] req@ffff954342f48e00 x1865013828674816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628712.226646:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828674816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628712.226659:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628712.226663:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628712.226668:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828674816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628712.417828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628712.417919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628713.185799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628713.441761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628713.441845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628714.209750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628714.465777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628714.465900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628715.233910:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628715.489797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628715.489919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628716.257815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628716.513843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628716.513948:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628717.281888:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628717.345960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628717.345977:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628717.345985:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628717.346001:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13051 00000100:00080000:3.0:1778628717.346020:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628717.346031:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628717.346086:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828674944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628717.346159:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628717.346198:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628717.346203:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628717.346213:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13051 00000100:00080000:3.0:1778628717.346224:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628717.346233:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628717.346254:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628717.346264:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828674944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628717.346289:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828675072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628717.346310:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628717.346315:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628717.346318:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628717.346323:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13051 00000100:00080000:3.0:1778628717.346329:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628717.346334:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628717.346342:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628717.346357:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828675200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628717.346371:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628717.346375:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628717.346378:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778628717.346387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628717.346388:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13051 00000100:00080000:3.0:1778628717.346398:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628717.346403:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628717.346422:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628717.346442:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828675328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628717.346457:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628717/real 1778628717] req@ffff95434701d500 x1865013828674944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628717.346459:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628717.346472:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628717.346477:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628717.346495:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13051 00000100:00080000:3.0:1778628717.346516:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628717.346525:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828674944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628717.346529:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628717.346546:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628717.346550:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628717.346575:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628717.346580:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828674944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628717.346586:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828675456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628717.346615:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628717.346638:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628717.346648:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628717/real 1778628717] req@ffff95434701c380 x1865013828675072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628717.346746:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828675072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628717.346760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628717.346764:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628717.346769:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628717.346778:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628717.346808:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628717.346822:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628717/real 1778628717] req@ffff95434701f100 x1865013828675200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628717.346837:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828675200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628717.346847:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628717.346851:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628717.346883:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628717.346907:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628717.346928:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628717.346937:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628717/real 1778628717] req@ffff95434701d880 x1865013828675328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628717.346947:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828675328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628717.346966:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628717.346969:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628717.346972:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628717.346977:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628717.346995:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628717.347003:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628717/real 1778628717] req@ffff95434701f800 x1865013828675456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628717.347012:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828675456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628717.347020:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628717.347023:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628717.347025:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628717.537894:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628717.538049:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628718.305758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628718.561799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628718.561898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628719.329915:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628719.585842:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628719.585926:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628720.353751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628720.609780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628720.609903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628721.377741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628721.633832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628721.633910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628722.401738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628722.465765:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628722.465775:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628722.465779:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628722.465787:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13056 00000100:00080000:3.0:1778628722.465808:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628722.465814:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628722.465846:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828675584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628722.465871:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628722.465873:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628722.465875:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628722.465883:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13056 00000100:00080000:3.0:1778628722.465900:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628722.465903:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628722.465911:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628722.465920:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828675584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628722.465924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828675712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628722.465934:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628722.465937:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628722.465938:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628722.465941:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13056 00000100:00080000:3.0:1778628722.465943:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628722.465945:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628722.465948:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628722.465956:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828675840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628722.465962:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628722.465965:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628722.465966:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628722.465969:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13056 00000100:00080000:3.0:1778628722.465971:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628722.465973:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628722.465976:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628722.465982:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628722.465985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828675968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628722.466000:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628722.466003:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628722.466005:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628722.466007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13056 00000100:00080000:3.0:1778628722.466010:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628722.466012:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628722.466016:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628722.466025:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628722/real 1778628722] req@ffff95434701c700 x1865013828675584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628722.466026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828676096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628722.466079:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828675584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628722.466110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628722.466115:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628722.466118:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828675584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628722.466148:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628722.466166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628722.466176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628722/real 1778628722] req@ffff95434701c000 x1865013828675712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628722.466184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828675712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628722.466191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628722.466193:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628722.466195:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628722.466199:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628722.466209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628722.466214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628722/real 1778628722] req@ffff95434701ea00 x1865013828675840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628722.466221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828675840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628722.466226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628722.466228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628722.466230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628722.466239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628722.466252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628722.466259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628722/real 1778628722] req@ffff95434701e680 x1865013828675968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628722.466266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828675968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628722.466271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628722.466273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628722.466275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828675968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628722.466278:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628722.466287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628722.466292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628722/real 1778628722] req@ffff95434701df80 x1865013828676096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628722.466300:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828676096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628722.466305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628722.466306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628722.466308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628722.657747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628722.657845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628723.425820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628723.681812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628723.681937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628724.449818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628724.705786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628724.705833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628725.473780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628725.729806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628725.729910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628726.497839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628726.753877:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628726.753990:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628727.521758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628727.585811:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628727.585820:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628727.585824:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628727.585832:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13062 00000100:00080000:0.0:1778628727.585843:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628727.585851:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628727.585893:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828676224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628727.585937:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628727.585940:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628727.585942:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628727.585945:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13062 00000100:00080000:0.0:1778628727.585948:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628727.585951:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628727.585958:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628727.585984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828676352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628727.585984:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828676224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628727.585998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628727.586000:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628727.586001:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628727.586005:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13062 00000100:00080000:0.0:1778628727.586008:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628727.586010:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628727.586014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628727.586024:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828676480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628727.586030:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628727.586033:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628727.586035:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628727.586037:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13062 00000100:00080000:0.0:1778628727.586039:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628727.586041:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628727.586044:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628727.586046:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628727.586054:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828676608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628727.586062:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628727.586064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628727.586065:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628727.586072:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13062 00000100:00080000:0.0:1778628727.586075:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628727.586080:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628727.586080:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628727/real 1778628727] req@ffff954342f4bb80 x1865013828676224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778628727.586085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628727.586097:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828676736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628727.586103:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828676224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628727.586115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628727.586119:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628727.586122:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828676224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628727.586140:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628727.586153:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628727.586160:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628727/real 1778628727] req@ffff954342f49880 x1865013828676352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628727.586168:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828676352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628727.586173:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628727.586175:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628727.586178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628727.586181:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628727.586190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628727.586195:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628727/real 1778628727] req@ffff954342f4a300 x1865013828676480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628727.586210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828676480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628727.586215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628727.586217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628727.586219:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628727.586223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628727.586238:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628727.586248:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628727/real 1778628727] req@ffff954342f4b480 x1865013828676608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628727.586260:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828676608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628727.586269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628727.586272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628727.586275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628727.586282:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628727.586300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628727.586308:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628727/real 1778628727] req@ffff954342f49c00 x1865013828676736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628727.586316:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828676736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628727.586331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628727.586333:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628727.586335:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628727.777794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628727.777900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628728.545749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628728.801826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628728.801873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628729.569785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628729.825780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628729.825869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628730.593818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628730.849786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628730.849897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628731.617794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628731.873803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628731.873890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628732.641765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628732.705800:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628732.705810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628732.705816:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628732.705828:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13067 00000100:00080000:1.0:1778628732.705843:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628732.705851:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628732.705897:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828676864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628732.705928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628732.705932:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628732.705934:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628732.705940:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13067 00000100:00080000:1.0:1778628732.705945:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628732.705949:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628732.705961:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628732.705966:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828676864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628732.705976:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828676992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628732.706006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628732.706010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628732.706012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628732.706016:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13067 00000100:00080000:1.0:1778628732.706029:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628732.706032:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628732.706038:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778628732.706039:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628732.706053:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828677120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628732.706063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628732.706067:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628732.706069:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628732.706074:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13067 00000100:00080000:1.0:1778628732.706078:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628732.706082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628732/real 1778628732] req@ffff954350197b80 x1865013828676864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628732.706085:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628732.706093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628732.706105:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828677248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628732.706106:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828676864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628732.706138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628732.706139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628732.706145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628732.706146:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778628732.706148:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778628732.706152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828676864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628732.706156:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13067 00000100:00080000:1.0:1778628732.706166:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628732.706170:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628732.706175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628732.706182:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628732.706188:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828677376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628732.706214:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628732.706230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628732/real 1778628732] req@ffff954350195180 x1865013828676992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628732.706261:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828676992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628732.706268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628732.706270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628732.706273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828676992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628732.706277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628732.706294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628732.706301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628732/real 1778628732] req@ffff954350195c00 x1865013828677120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628732.706310:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828677120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628732.706315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628732.706317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628732.706319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628732.706323:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628732.706333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628732.706339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628732/real 1778628732] req@ffff954350195500 x1865013828677248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628732.706346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828677248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628732.706360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628732.706363:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628732.706365:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628732.706377:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628732.706390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628732.706397:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628732/real 1778628732] req@ffff954350195f80 x1865013828677376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628732.706404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828677376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628732.706411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628732.706413:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628732.706415:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628732.897769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628732.897868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628733.665744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628733.921890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628733.922077:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628734.689761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628734.946592:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778628734.946623:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628735.713764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628735.969920:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628735.971095:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628736.737719:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628736.993848:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628736.993909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628737.761795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628737.825934:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628737.825946:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628737.825953:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628737.825964:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13072 00000100:00080000:1.0:1778628737.825978:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628737.825987:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628737.826048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828677504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628737.826083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628737.826088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628737.826091:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628737.826096:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13072 00000100:00080000:1.0:1778628737.826100:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628737.826105:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628737.826116:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628737.826134:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828677632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628737.826160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828677504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628737.826190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628737.826195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628737.826198:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628737.826205:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13072 00000100:00080000:1.0:1778628737.826212:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628737.826217:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628737.826226:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628737.826241:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828677760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628737.826254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628737.826264:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1778628737.826265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628737.826270:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628737.826279:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13072 00000100:00080000:1.0:1778628737.826286:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628737.826291:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628737.826298:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628737.826314:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828677888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628737.826325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628737/real 1778628737] req@ffff954350195880 x1865013828677504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628737.826333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628737.826348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628737.826354:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628737.826365:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13072 00000100:00100000:0.0:1778628737.826377:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828677504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628737.826381:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628737.826388:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628737.826399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778628737.826420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778628737.826424:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828678016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628737.826472:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628737.826480:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828677504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628737.826515:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628737.826549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628737.826565:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628737/real 1778628737] req@ffff954350195f80 x1865013828677632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628737.826582:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828677632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628737.826592:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628737.826597:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628737.826600:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628737.826608:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628737.826626:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628737.826636:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628737/real 1778628737] req@ffff954350195500 x1865013828677760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628737.826650:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828677760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628737.826660:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628737.826663:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628737.826667:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628737.826727:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628737.826757:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628737.826778:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628737/real 1778628737] req@ffff954350195c00 x1865013828677888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628737.826801:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828677888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628737.826817:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628737.826823:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628737.826830:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828677888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628737.826842:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628737.826880:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628737.826901:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628737/real 1778628737] req@ffff954350195180 x1865013828678016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628737.826916:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828678016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628737.826927:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628737.826931:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628737.826935:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678016:192.168.204.102@tcp:8:kworker.0 00010000:00010000:2.0:1778628738.018138:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778628738.018179:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628738.785836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628739.041803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628739.041852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628739.809800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628740.065801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628740.065889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628740.833782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628741.089817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628741.089902:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628741.857812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628742.113770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628742.113881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628742.881767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628742.945733:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628742.945742:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628742.945745:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628742.945753:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13077 00000100:00080000:1.0:1778628742.945763:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628742.945768:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628742.945802:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828678144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628742.945823:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628742.945826:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628742.945827:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628742.945830:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13077 00000100:00080000:1.0:1778628742.945833:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628742.945835:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628742.945843:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628742.945851:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828678272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628742.945859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628742.945862:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828678144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628742.945876:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628742.945878:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628742.945882:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13077 00000100:00080000:1.0:1778628742.945885:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628742.945888:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628742.945892:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628742.945902:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828678400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628742.945909:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628742.945911:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628742.945912:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628742.945914:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13077 00000100:00080000:1.0:1778628742.945917:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628742.945921:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628742.945923:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628742.945928:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628742.945937:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828678528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628742.945943:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628742.945945:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628742.945947:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628742.945949:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13077 00000100:00080000:1.0:1778628742.945952:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628742.945954:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628742.945957:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628742/real 1778628742] req@ffff954350197b80 x1865013828678144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628742.945965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628742.945979:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828678656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628742.945982:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828678144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628742.946002:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628742.946005:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628742.946009:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828678144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628742.946028:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628742.946043:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628742.946051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628742/real 1778628742] req@ffff954350194e00 x1865013828678272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628742.946059:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828678272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628742.946065:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628742.946068:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628742.946070:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628742.946073:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628742.946083:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628742.946088:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628742/real 1778628742] req@ffff954350196a00 x1865013828678400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628742.946103:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828678400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628742.946108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628742.946110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628742.946112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628742.946116:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628742.946129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628742.946134:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628742/real 1778628742] req@ffff954350194a80 x1865013828678528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628742.946141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828678528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628742.946146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628742.946148:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628742.946150:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628742.946153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628742.946162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628742.946167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628742/real 1778628742] req@ffff954350194380 x1865013828678656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628742.946174:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828678656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628742.946188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628742.946190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628742.946192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628743.137804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628743.137878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628743.905773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628744.161779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628744.161881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628744.929749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628745.185763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628745.185814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628745.953754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628746.209747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628746.209857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628746.977797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628747.233764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628747.233873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628748.001730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628748.065862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628748.065878:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628748.065885:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628748.065901:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13082 00000100:00080000:2.0:1778628748.065932:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628748.065941:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628748.065994:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828678784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628748.066027:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628748.066031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628748.066033:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628748.066038:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13082 00000100:00080000:2.0:1778628748.066041:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628748.066044:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628748.066054:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628748.066066:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828678912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628748.066076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628748.066079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628748.066099:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628748.066103:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13082 00000100:00080000:2.0:1778628748.066108:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628748.066111:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628748.066118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628748.066132:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828679040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628748.066139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828678784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628748.066159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628748.066164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628748.066166:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628748.066172:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13082 00000100:00080000:2.0:1778628748.066175:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628748.066181:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628748.066186:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628748.066197:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828679168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628748.066208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628748.066229:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628748.066231:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628748.066234:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13082 00000100:00080000:2.0:1778628748.066242:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628748.066293:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628748.066295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778628748.066303:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628748.066326:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828679296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628748.066427:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628748/real 1778628748] req@ffff954358789c00 x1865013828678784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628748.066461:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828678784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628748.066480:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628748.066504:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628748.066511:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828678784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628748.066525:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628748.066574:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628748.066595:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628748/real 1778628748] req@ffff954358788e00 x1865013828678912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628748.066615:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828678912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628748.066645:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628748.066652:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628748.066658:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828678912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628748.066694:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628748.066773:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628748.066787:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628748/real 1778628748] req@ffff954358788700 x1865013828679040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628748.066800:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828679040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628748.066808:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628748.066812:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628748.066815:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628748.066831:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628748.066847:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628748.066856:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628748/real 1778628748] req@ffff954358789880 x1865013828679168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628748.066868:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828679168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628748.066875:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628748.066878:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628748.066889:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628748.066896:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628748.066915:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628748.066925:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628748/real 1778628748] req@ffff954358789500 x1865013828679296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628748.066936:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828679296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628748.066945:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628748.066948:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628748.066951:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628748.257810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628748.257985:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628749.025757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628749.281736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628749.281903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628750.049726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628750.305730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628750.305831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628751.073792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628751.329753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628751.329893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628752.097754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628752.353825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628752.353951:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628753.121759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628753.185828:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628753.185861:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628753.185869:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628753.185888:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13087 00000100:00080000:2.0:1778628753.185909:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628753.185921:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628753.186013:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828679424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628753.186062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628753.186079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628753.186082:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628753.186090:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13087 00000100:00080000:2.0:1778628753.186097:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628753.186104:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628753.186121:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628753.186177:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828679424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628753.186178:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828679552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628753.186207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628753.186244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628753.186248:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628753.186257:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13087 00000100:00080000:2.0:1778628753.186267:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628753.186275:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628753.186285:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778628753.186301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628753.186314:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828679680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628753.186349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628753.186355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1778628753.186358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628753/real 1778628753] req@ffff95435878b800 x1865013828679424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628753.186360:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628753.186379:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13087 00000100:00080000:2.0:1778628753.186388:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628753.186395:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828679424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628753.186400:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628753.186419:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:0.0:1778628753.186422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628753.186432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778628753.186448:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828679808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628753.186454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828679424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628753.186480:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628753.186486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628753.186490:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628753.186499:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13087 00000100:00100000:0.0:1778628753.186501:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679552:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778628753.186512:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628753.186522:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628753.186542:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778628753.186551:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628753.186560:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628753/real 1778628753] req@ffff95435878a300 x1865013828679552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628753.186582:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828679552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628753.186585:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828679936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628753.186611:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628753.186623:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628753.186630:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628753.186640:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628753.186661:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628753.186712:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628753/real 1778628753] req@ffff95435878ad80 x1865013828679680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628753.186727:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828679680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628753.186736:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628753.186740:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628753.186743:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628753.186749:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628753.186788:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628753.186811:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628753/real 1778628753] req@ffff954358789180 x1865013828679808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628753.186825:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828679808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628753.186834:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628753.186838:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628753.186841:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628753.186865:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628753.186884:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628753.186894:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628753/real 1778628753] req@ffff954358788a80 x1865013828679936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628753.186906:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828679936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628753.186916:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628753.186919:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628753.186924:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828679936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628753.377775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628753.377943:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628754.145799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628754.401753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628754.401844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628755.169778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628755.425823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628755.425962:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628756.193748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628756.449748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628756.449914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628757.217806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628757.473855:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628757.473992:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628758.241770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628758.305828:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628758.305837:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628758.305841:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628758.305849:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13092 00000100:00080000:1.0:1778628758.305859:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628758.305865:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628758.305898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828680064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628758.305928:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628758.305931:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628758.305932:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628758.305936:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13092 00000100:00080000:1.0:1778628758.305938:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628758.305940:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628758.305948:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628758.305957:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828680192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628758.305963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628758.305966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628758.305967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628758.305969:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13092 00000100:00080000:1.0:1778628758.305986:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628758.305990:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628758.305994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628758.306002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828680320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628758.306008:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828680064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628758.306012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628758.306015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628758.306016:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628758.306021:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13092 00000100:00080000:1.0:1778628758.306024:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628758.306029:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628758.306034:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628758.306060:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828680448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628758.306071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628758.306075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628758.306077:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628758.306080:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13092 00000100:00080000:1.0:1778628758.306101:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628758.306102:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628758.306107:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628758.306115:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628758.306127:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828680576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628758.306148:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628758/real 1778628758] req@ffff954350194700 x1865013828680064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628758.306165:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828680064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628758.306174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628758.306178:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628758.306181:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828680064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628758.306186:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628758.306202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628758.306209:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628758/real 1778628758] req@ffff954350197800 x1865013828680192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628758.306217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828680192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628758.306223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628758.306225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628758.306228:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628758.306246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628758.306256:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628758.306262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628758/real 1778628758] req@ffff954350197480 x1865013828680320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628758.306269:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828680320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628758.306284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628758.306286:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628758.306288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628758.306292:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628758.306307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628758.306317:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628758/real 1778628758] req@ffff954350196d80 x1865013828680448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628758.306346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828680448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628758.306354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628758.306357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628758.306360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628758.306366:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628758.306386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628758.306397:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628758/real 1778628758] req@ffff954350194000 x1865013828680576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628758.306409:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828680576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628758.306417:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628758.306419:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628758.306426:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628758.497745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628758.497937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628759.265772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628759.521747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628759.521820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628760.289747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628760.545851:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628760.545983:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628761.313826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628761.569768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628761.569825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628762.337763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628762.593761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628762.593859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628763.361765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628763.425777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628763.425786:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628763.425790:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628763.425799:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13097 00000100:00080000:1.0:1778628763.425816:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628763.425822:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628763.425858:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828680704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628763.425882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628763.425886:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628763.425887:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628763.425890:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13097 00000100:00080000:1.0:1778628763.425893:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628763.425896:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628763.425905:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628763.425917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828680832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628763.425925:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628763.425928:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628763.425929:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628763.425933:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13097 00000100:00080000:1.0:1778628763.425935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628763.425938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628763.425970:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828680704:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778628763.425985:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628763.425999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828680960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628763.426008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628763.426011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628763.426013:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628763.426017:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13097 00000100:00080000:1.0:1778628763.426021:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628763.426024:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628763.426028:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628763.426037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828681088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628763.426044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628763.426047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628763.426048:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628763.426051:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13097 00000100:00080000:1.0:1778628763.426054:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628763.426060:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628763.426060:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628763.426067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628763.426076:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828681216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628763.426109:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628763/real 1778628763] req@ffff954350197100 x1865013828680704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628763.426147:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828680704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628763.426161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628763.426166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628763.426171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828680704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628763.426201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628763.426246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628763.426262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628763/real 1778628763] req@ffff954350194000 x1865013828680832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628763.426276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828680832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628763.426302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628763.426305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628763.426315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628763.426349:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628763.426390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628763.426410:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628763/real 1778628763] req@ffff954350196d80 x1865013828680960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628763.426432:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828680960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628763.426446:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628763.426451:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628763.426458:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828680960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628763.426469:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628763.426518:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628763.426531:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628763/real 1778628763] req@ffff954350197480 x1865013828681088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628763.426546:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828681088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628763.426556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628763.426560:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628763.426564:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628763.426571:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628763.426606:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628763.426633:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628763/real 1778628763] req@ffff954350197800 x1865013828681216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628763.426643:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828681216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628763.426650:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628763.426653:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628763.426655:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628763.617772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628763.617902:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628764.385773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628764.641793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628764.641950:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628765.409760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628765.665789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628765.665998:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628766.433746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628766.689790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628766.689876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628767.457742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628767.713746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628767.713806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628768.481770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628768.545832:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628768.545970:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628768.545974:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628768.545982:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13103 00000100:00080000:3.0:1778628768.545991:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628768.545997:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628768.546035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828681344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628768.546071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628768.546075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628768.546078:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628768.546083:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13103 00000100:00080000:3.0:1778628768.546086:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628768.546104:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628768.546113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628768.546118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828681344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628768.546127:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828681472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628768.546136:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628768.546139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628768.546141:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628768.546143:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13103 00000100:00080000:3.0:1778628768.546147:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628768.546150:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628768.546159:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628768.546167:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828681600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628768.546174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628768.546177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628768.546178:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628768.546181:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13103 00000100:00080000:3.0:1778628768.546183:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628768.546186:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628768.546207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628768.546220:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828681728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628768.546230:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628768.546232:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681472:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778628768.546242:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628768.546244:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628768.546250:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13103 00000100:00080000:3.0:1778628768.546270:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628768.546274:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628768.546279:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628768.546284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778628768.546290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828681856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628768.546313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628768.546329:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628768.737821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628768.737933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628769.505793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628769.761840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628769.761947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628770.529788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628770.785787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628770.785865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628771.553845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778628771.618123:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778628771.618202:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778628771.618228:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628771.618281:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628771.618291:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628771.618301:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628771.618313:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628771.618393:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628768/real 1778628771] req@ffff95434701df80 x1865013828681728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628771.618440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828681728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628771.618487:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628771.618496:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628771.618504:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628771.618517:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628768/real 1778628771] req@ffff95434701e680 x1865013828681856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628771.618541:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828681856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628771.618557:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628771.618563:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628771.618569:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628771.618581:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628768/real 1778628771] req@ffff95434701f480 x1865013828681472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628771.618605:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828681472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628771.618622:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628771.618628:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628771.618634:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628771.618646:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628768/real 1778628771] req@ffff95434701ca80 x1865013828681600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628771.618665:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828681600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628771.618740:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628771.618748:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628771.618754:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828681600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628771.618767:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628768/real 1778628771] req@ffff95434701d180 x1865013828681344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628771.618818:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828681344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628771.618833:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628771.618839:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628771.618845:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828681344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628771.809836:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628771.809947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628772.577850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628772.833791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628772.833849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628773.601867:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628773.665764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628773.665775:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628773.665779:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628773.665791:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13108 00000100:00080000:3.0:1778628773.665803:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628773.665810:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628773.665845:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828681984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628773.665875:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628773.665879:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628773.665881:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628773.665886:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13108 00000100:00080000:3.0:1778628773.665903:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628773.665906:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628773.665914:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628773.665925:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828682112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628773.665936:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628773.665940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628773.665942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628773.665947:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13108 00000100:00080000:3.0:1778628773.665951:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628773.665962:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628773.665969:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628773.665983:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828682240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628773.665998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628773.666003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628773.666004:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628773.666015:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13108 00000100:00080000:3.0:1778628773.666026:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628773.666030:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828681984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628773.666036:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628773.666050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628773.666065:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828682368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628773.666087:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628773.666091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628773.666093:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628773.666098:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13108 00000100:00080000:3.0:1778628773.666102:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628773.666107:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628773.666113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628773.666128:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828682496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628773.666200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628773.666280:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628773/real 1778628773] req@ffff95434701ea00 x1865013828681984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628773.666323:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828681984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628773.666347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628773.666357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628773.666366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828681984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628773.666417:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628773.666471:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628773.666499:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628773/real 1778628773] req@ffff95434701c000 x1865013828682112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628773.666521:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828682112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628773.666538:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628773.666545:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628773.666552:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628773.666605:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628773.666647:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628773.666799:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628773/real 1778628773] req@ffff95434701c700 x1865013828682240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628773.666838:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828682240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628773.666854:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628773.666864:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628773.666871:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628773.666893:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628773.666938:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628773.666964:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628773/real 1778628773] req@ffff95434701f800 x1865013828682368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628773.667004:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828682368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628773.667021:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628773.667028:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628773.667035:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628773.667048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628773.667108:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628773.667127:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628773/real 1778628773] req@ffff95434701d880 x1865013828682496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628773.667147:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828682496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628773.667163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628773.667168:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628773.667173:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628773.857769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628773.857930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628774.625790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628774.881749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628774.881804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628775.649762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628775.905755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628775.905832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628776.673812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628776.929830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628776.929861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628777.697839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628777.953840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628777.953937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628778.721793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628778.785849:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628778.785868:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628778.785876:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628778.785909:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13113 00000100:00080000:3.0:1778628778.785930:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628778.785942:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628778.786015:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828682624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628778.786074:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628778.786078:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628778.786103:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628778.786108:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13113 00000100:00080000:3.0:1778628778.786114:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628778.786118:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628778.786130:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628778.786144:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828682752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628778.786155:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628778.786159:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628778.786161:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628778.786165:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13113 00000100:00080000:3.0:1778628778.786168:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628778.786172:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628778.786178:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628778.786188:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828682624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628778.786189:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828682880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628778.786207:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628778.786210:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628778.786212:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628778.786216:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13113 00000100:00080000:3.0:1778628778.786219:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628778.786223:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628778.786227:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628778.786251:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828683008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628778.786261:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628778.786264:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628778.786266:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628778.786270:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13113 00000100:00080000:3.0:1778628778.786274:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628778.786278:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628778.786282:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778628778.786289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628778.786297:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828683136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628778.786344:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628778/real 1778628778] req@ffff95434701f100 x1865013828682624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628778.786396:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828682624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628778.786411:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628778.786416:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628778.786421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828682624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628778.786430:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628778.786457:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628778.786469:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628778/real 1778628778] req@ffff95434701c380 x1865013828682752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628778.786482:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828682752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628778.786492:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628778.786495:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628778.786499:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628778.786529:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628778.786545:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628778.786554:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628778/real 1778628778] req@ffff95434701d500 x1865013828682880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628778.786566:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828682880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628778.786589:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628778.786592:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628778.786596:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828682880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628778.786603:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628778.786624:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628778.786634:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628778/real 1778628778] req@ffff95434701d880 x1865013828683008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628778.786711:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828683008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628778.786721:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628778.786725:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628778.786728:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628778.786735:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628778.786764:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628778.786775:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628778/real 1778628778] req@ffff95434701f800 x1865013828683136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628778.786787:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828683136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628778.786796:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628778.786800:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628778.786812:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628778.977782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628778.977880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628779.745822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628780.001837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628780.001903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628780.769788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628781.025788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628781.025957:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628781.793784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628782.049773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628782.049837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628782.817801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628783.073814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628783.073938:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628783.841800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628783.905893:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628783.905906:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628783.905912:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628783.905924:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13118 00000100:00080000:0.0:1778628783.905940:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628783.905948:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628783.906010:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828683264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628783.906048:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628783.906051:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628783.906053:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628783.906057:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13118 00000100:00080000:0.0:1778628783.906060:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628783.906063:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628783.906073:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628783.906084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828683392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628783.906102:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628783.906129:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628783.906131:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628783.906134:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13118 00000100:00080000:0.0:1778628783.906138:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628783.906184:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628783.906191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828683264:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778628783.906193:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628783.906214:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828683520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628783.906246:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628783.906256:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628783.906258:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628783.906263:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13118 00000100:00080000:0.0:1778628783.906269:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628783.906273:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628783.906278:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628783.906291:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828683648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628783.906301:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628783.906305:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628783.906307:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628783.906311:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13118 00000100:00080000:0.0:1778628783.906326:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628783.906330:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778628783.906340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778628783.906361:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628783.906383:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828683776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628783.906442:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628783/real 1778628783] req@ffff954342f48700 x1865013828683264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628783.906483:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828683264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628783.906504:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628783.906527:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628783.906533:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828683264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628783.906562:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628783.906589:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628783.906605:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628783/real 1778628783] req@ffff954342f4b100 x1865013828683392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628783.906621:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828683392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628783.906632:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628783.906637:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628783.906641:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628783.906773:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628783.906803:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628783.906817:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628783/real 1778628783] req@ffff954342f49f80 x1865013828683520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628783.906849:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828683520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628783.906858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628783.906862:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628783.906865:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628783.906873:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628783.906902:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628783.906918:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628783/real 1778628783] req@ffff954342f49500 x1865013828683648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628783.906938:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828683648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628783.906953:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628783.906958:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628783.906964:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628783.906990:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628783.907014:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628783.907024:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628783/real 1778628783] req@ffff954342f4ad80 x1865013828683776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628783.907034:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828683776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628783.907052:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628783.907055:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628783.907057:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828683776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628784.097838:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628784.098051:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628784.865789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628785.121771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628785.121874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628785.889793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628786.145782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628786.145843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628786.913752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628787.169754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628787.169841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628787.937779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628788.193792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628788.193880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628788.961851:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628789.025797:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628789.025812:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628789.025818:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628789.025831:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13123 00000100:00080000:3.0:1778628789.025844:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628789.025875:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628789.025926:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828683904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628789.025955:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628789.025958:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628789.025959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628789.025963:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13123 00000100:00080000:3.0:1778628789.025966:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628789.025969:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628789.025977:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628789.025986:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828684032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628789.025993:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628789.025996:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628789.025997:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628789.025999:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13123 00000100:00080000:3.0:1778628789.026000:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628789.026002:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628789.026005:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628789.026010:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828684160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628789.026018:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628789.026021:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628789.026026:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628789.026030:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13123 00000100:00080000:3.0:1778628789.026033:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628789.026036:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628789.026041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628789.026044:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828683904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628789.026052:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828684288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628789.026071:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628789.026073:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628789.026075:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628789.026078:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13123 00000100:00080000:3.0:1778628789.026080:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628789.026082:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628789.026085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628789.026095:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828684416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628789.026128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628789.026175:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628789/real 1778628789] req@ffff95434701c700 x1865013828683904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628789.026191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828683904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628789.026200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628789.026203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628789.026206:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828683904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628789.026211:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628789.026225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628789.026232:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628789/real 1778628789] req@ffff95434701c000 x1865013828684032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628789.026240:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828684032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628789.026246:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628789.026248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628789.026250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628789.026262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628789.026276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628789.026282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628789/real 1778628789] req@ffff95434701ea00 x1865013828684160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628789.026290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828684160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628789.026295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628789.026296:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628789.026298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628789.026321:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628789.026332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628789.026338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628789/real 1778628789] req@ffff95434701d180 x1865013828684288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628789.026346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828684288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628789.026351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628789.026353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628789.026355:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628789.026358:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628789.026368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628789.026373:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628789/real 1778628789] req@ffff95434701ca80 x1865013828684416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628789.026386:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828684416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628789.026392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628789.026395:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628789.026397:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628789.217745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628789.217832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628789.985811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628790.241778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628790.241881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628791.009719:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628791.265749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628791.265842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628792.033748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628792.289766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628792.289836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628793.057759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628793.313765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628793.313811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628794.081755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628794.145789:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628794.145799:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628794.145804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628794.145813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13128 00000100:00080000:1.0:1778628794.145823:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628794.145829:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628794.145862:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828684544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628794.145887:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628794.145890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628794.145892:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628794.145894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13128 00000100:00080000:1.0:1778628794.145897:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628794.145899:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628794.145907:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628794.145916:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828684672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628794.145933:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828684544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628794.145949:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628794.145969:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628794.145971:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628794.145975:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13128 00000100:00080000:1.0:1778628794.145980:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628794.145983:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628794.145987:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628794.146001:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828684800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628794.146009:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628794.146012:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628794.146013:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628794.146016:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13128 00000100:00080000:1.0:1778628794.146043:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628794.146046:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628794.146050:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628794.146055:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628794.146064:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828684928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628794.146073:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628794.146076:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628794.146077:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628794.146080:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13128 00000100:00080000:1.0:1778628794.146090:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628794.146094:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628794.146099:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628794/real 1778628794] req@ffff954350194700 x1865013828684544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628794.146099:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628794.146126:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828685056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628794.146132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828684544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628794.146149:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628794.146154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628794.146159:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828684544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628794.146192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628794.146210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628794.146252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628794/real 1778628794] req@ffff954350194380 x1865013828684672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628794.146320:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828684672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628794.146330:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628794.146370:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628794.146383:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628794.146388:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628794.146404:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628794.146411:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628794/real 1778628794] req@ffff954350194a80 x1865013828684800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628794.146435:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828684800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628794.146451:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628794.146454:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628794.146456:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628794.146460:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628794.146473:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628794.146479:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628794/real 1778628794] req@ffff954350196a00 x1865013828684928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628794.146488:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828684928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628794.146504:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628794.146507:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628794.146509:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828684928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628794.146513:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628794.146549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628794.146568:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628794/real 1778628794] req@ffff954350194e00 x1865013828685056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628794.146580:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828685056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628794.146605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628794.146607:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628794.146610:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628794.337817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628794.337970:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628795.105810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628795.361741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628795.361808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628796.129752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628796.385741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628796.385805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628797.153810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628797.409744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628797.409820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628798.177853:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628798.433792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628798.433940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628799.201788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628799.265811:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628799.265823:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628799.265828:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628799.265839:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13133 00000100:00080000:3.0:1778628799.265853:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628799.265859:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628799.265924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828685184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628799.265954:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628799.265957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628799.265958:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628799.265961:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13133 00000100:00080000:3.0:1778628799.265964:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628799.265967:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628799.265983:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628799.265993:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828685184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628799.265996:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828685312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628799.266008:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628799.266011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628799.266012:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628799.266015:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13133 00000100:00080000:3.0:1778628799.266017:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628799.266019:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628799.266023:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628799.266051:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828685440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628799.266058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628799.266061:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628799.266062:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628799.266065:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13133 00000100:00080000:3.0:1778628799.266068:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628799.266070:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628799.266074:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628799.266079:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628799.266086:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828685568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628799.266093:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628799.266096:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628799.266098:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628799.266101:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13133 00000100:00080000:3.0:1778628799.266103:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628799.266106:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628799.266109:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628799.266117:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828685696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628799.266127:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628799/real 1778628799] req@ffff95434701f480 x1865013828685184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628799.266146:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828685184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628799.266156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628799.266160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628799.266163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828685184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628799.266187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628799.266205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628799.266213:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628799/real 1778628799] req@ffff95434701e680 x1865013828685312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628799.266222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828685312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628799.266228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628799.266230:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628799.266253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628799.266258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628799.266292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628799.266302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628799/real 1778628799] req@ffff95434701df80 x1865013828685440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628799.266312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828685440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628799.266319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628799.266321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628799.266324:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628799.266328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628799.266342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628799.266348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628799/real 1778628799] req@ffff95434701ca80 x1865013828685568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628799.266357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828685568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628799.266389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628799.266392:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628799.266395:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628799.266399:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628799.266414:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628799.266429:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628799/real 1778628799] req@ffff95434701d180 x1865013828685696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628799.266456:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828685696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628799.266464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628799.266467:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628799.266469:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628799.457841:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628799.457953:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628800.225782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628800.481800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628800.481928:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628801.249755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628801.505962:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778628801.505986:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628802.273722:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628802.529840:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628802.529933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628803.297726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628803.554244:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628803.554342:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628804.321749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628804.385768:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628804.385778:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628804.385784:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628804.385795:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13138 00000100:00080000:1.0:1778628804.385808:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628804.385817:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628804.385859:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828685824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628804.385890:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628804.385893:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628804.385895:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628804.385898:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13138 00000100:00080000:1.0:1778628804.385901:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628804.385905:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628804.385914:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628804.385939:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828685824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628804.385949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828685952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628804.385981:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628804.385986:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628804.385988:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628804.385994:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13138 00000100:00080000:1.0:1778628804.386000:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628804.386006:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628804.386012:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628804.386026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828686080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778628804.386032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628804.386053:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628804.386057:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628804.386060:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628804.386065:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13138 00000100:00080000:1.0:1778628804.386070:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628804.386073:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628804/real 1778628804] req@ffff954350197b80 x1865013828685824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628804.386077:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628804.386085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628804.386102:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828686208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628804.386111:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828685824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628804.386115:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628804.386125:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628804.386129:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628804.386154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628804.386156:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13138 00000100:00080000:1.0:1778628804.386165:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628804.386167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628804.386185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828685824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628804.386189:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628804.386196:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628804.386209:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828686336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628804.386214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628804.386254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628804.386269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628804/real 1778628804] req@ffff954350195180 x1865013828685952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628804.386280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828685952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628804.386288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628804.386291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628804.386293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828685952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628804.386298:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628804.386311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628804.386318:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628804/real 1778628804] req@ffff954350195c00 x1865013828686080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628804.386339:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828686080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628804.386346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628804.386349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628804.386353:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628804.386361:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628804.386387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628804.386401:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628804/real 1778628804] req@ffff954350195500 x1865013828686208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628804.386417:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828686208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628804.386428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628804.386432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628804.386435:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628804.386461:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628804.386481:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628804.386496:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628804/real 1778628804] req@ffff954350195f80 x1865013828686336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628804.386505:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828686336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628804.386523:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628804.386525:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628804.386528:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628804.577779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628804.577840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628805.345748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628805.601845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628805.602069:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628806.369739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628806.625781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628806.625835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628807.393790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628807.649746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628807.649860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628808.417746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628808.673826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628808.673966:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628809.441731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628809.505759:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628809.505768:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628809.505772:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628809.505781:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13144 00000100:00080000:1.0:1778628809.505791:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628809.505797:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628809.505866:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828686464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628809.505920:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628809.505924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628809.505926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628809.505930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13144 00000100:00080000:1.0:1778628809.505934:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628809.505937:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628809.505947:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628809.505974:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828686592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628809.506000:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828686464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628809.506048:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628809.506052:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628809.506054:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628809.506058:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13144 00000100:00080000:1.0:1778628809.506063:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628809.506067:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628809.506072:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628809.506096:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828686720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628809.506106:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628809.506108:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628809.506110:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628809.506113:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13144 00000100:00080000:1.0:1778628809.506117:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628809.506119:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628809.506124:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628809.506130:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628809.506135:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828686848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628809.506145:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628809.506148:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628809.506150:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628809.506153:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13144 00000100:00080000:1.0:1778628809.506164:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628809.506169:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628809.506173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628809/real 1778628809] req@ffff954350195880 x1865013828686464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778628809.506193:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628809.506208:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828686976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628809.506217:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828686464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628809.506228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628809.506234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628809.506240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828686464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628809.506298:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628809.506331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628809.506349:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628809/real 1778628809] req@ffff954350195f80 x1865013828686592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628809.506366:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828686592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628809.506407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628809.506412:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628809.506416:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628809.506425:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628809.506445:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628809.506457:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628809/real 1778628809] req@ffff954350195500 x1865013828686720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628809.506471:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828686720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628809.506480:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628809.506484:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628809.506488:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628809.506496:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628809.506520:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628809.506530:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628809/real 1778628809] req@ffff954350195c00 x1865013828686848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628809.506540:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828686848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628809.506546:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628809.506548:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628809.506569:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628809.506577:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628809.506601:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628809.506614:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628809/real 1778628809] req@ffff954350195180 x1865013828686976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628809.506628:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828686976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628809.506637:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628809.506640:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628809.506643:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828686976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628809.697754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628809.697820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628810.465753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628810.721731:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628810.721835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628811.489741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628811.745750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628811.745823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628812.513748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628812.769760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628812.769836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628813.537780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628813.793756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628813.793854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628814.561753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628814.625774:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628814.625783:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628814.625787:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628814.625795:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13149 00000100:00080000:3.0:1778628814.625804:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628814.625810:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628814.625849:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828687104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628814.625879:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628814.625882:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628814.625883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628814.625887:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13149 00000100:00080000:3.0:1778628814.625890:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628814.625893:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628814.625901:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628814.625911:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828687232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628814.625950:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628814.625953:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628814.625955:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628814.625958:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13149 00000100:00080000:3.0:1778628814.625962:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628814.625966:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628814.625980:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628814.625988:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828687104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628814.625991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828687360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628814.626002:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628814.626004:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628814.626005:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628814.626007:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13149 00000100:00080000:3.0:1778628814.626010:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628814.626012:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628814.626015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628814.626023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828687488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628814.626030:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628814.626032:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628814.626033:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628814.626035:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13149 00000100:00080000:3.0:1778628814.626045:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628814.626050:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628814.626054:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778628814.626059:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628814.626075:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828687616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628814.626087:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628814/real 1778628814] req@ffff95434701ea00 x1865013828687104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628814.626112:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828687104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628814.626120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628814.626133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628814.626143:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828687104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628814.626150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628814.626173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628814.626186:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628814/real 1778628814] req@ffff95434701c000 x1865013828687232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628814.626199:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828687232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628814.626208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628814.626212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628814.626216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628814.626236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628814.626264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628814.626276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628814/real 1778628814] req@ffff95434701c700 x1865013828687360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628814.626288:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828687360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628814.626297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628814.626300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628814.626304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628814.626310:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628814.626325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628814.626339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628814/real 1778628814] req@ffff95434701f800 x1865013828687488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628814.626347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828687488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628814.626356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628814.626359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628814.626362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628814.626368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628814.626387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628814.626398:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628814/real 1778628814] req@ffff95434701d880 x1865013828687616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628814.626411:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828687616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628814.626417:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628814.626419:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628814.626421:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628814.817738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628814.817785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628815.585786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628815.841744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628815.841859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628816.609758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628816.865830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628816.865940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628817.633828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628817.889830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628817.889904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628818.657815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628818.913861:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628818.913936:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628819.681818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628819.745912:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628819.745925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628819.745930:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628819.745941:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13154 00000100:00080000:2.0:1778628819.745954:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628819.745960:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628819.746003:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828687744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628819.746040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628819.746043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628819.746045:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628819.746049:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13154 00000100:00080000:2.0:1778628819.746051:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628819.746054:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628819.746063:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628819.746102:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828687872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628819.746110:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628819.746112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628819.746114:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628819.746117:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13154 00000100:00080000:2.0:1778628819.746120:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778628819.746123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828687744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628819.746125:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628819.746132:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628819.746139:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828688000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628819.746146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628819.746148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628819.746149:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628819.746153:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13154 00000100:00080000:2.0:1778628819.746156:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628819.746159:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628819.746163:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628819.746170:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828688128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628819.746176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628819.746179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628819.746180:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628819.746187:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13154 00000100:00080000:2.0:1778628819.746190:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628819.746193:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628819.746197:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:0.0:1778628819.746198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628819.746207:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828688256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628819.746246:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628819/real 1778628819] req@ffff95435878a680 x1865013828687744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628819.746263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828687744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628819.746272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628819.746275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628819.746279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828687744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628819.746283:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628819.746298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628819.746306:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628819/real 1778628819] req@ffff954358789f80 x1865013828687872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628819.746314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828687872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628819.746320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628819.746322:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628819.746324:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828687872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628819.746347:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628819.746357:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628819.746363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628819/real 1778628819] req@ffff954358788a80 x1865013828688000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628819.746370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828688000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628819.746384:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628819.746386:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628819.746388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628819.746392:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628819.746405:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628819.746411:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628819/real 1778628819] req@ffff954358789180 x1865013828688128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628819.746418:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828688128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628819.746424:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628819.746425:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628819.746427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628819.746431:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628819.746441:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628819.746446:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628819/real 1778628819] req@ffff95435878ad80 x1865013828688256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628819.746454:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828688256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628819.746459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628819.746461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628819.746467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628819.937798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628819.938012:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628820.705770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628820.961753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628820.961825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628821.729772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628821.985776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628821.985860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628822.753761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628823.009790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628823.009860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628823.777805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628824.033812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628824.033960:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628824.801771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628824.865851:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628824.865861:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628824.865866:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628824.865876:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13159 00000100:00080000:3.0:1778628824.865890:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628824.865897:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628824.865942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828688384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628824.865990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628824.865994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628824.865995:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628824.866021:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13159 00000100:00080000:3.0:1778628824.866026:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628824.866031:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628824.866048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828688384:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778628824.866065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628824.866086:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828688512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628824.866115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628824.866118:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628824.866119:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628824.866122:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13159 00000100:00080000:3.0:1778628824.866126:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628824.866129:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628824.866134:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778628824.866136:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628824.866150:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828688640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628824.866158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628824.866162:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628824.866180:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628824.866189:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13159 00000100:00100000:0.0:1778628824.866190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628824/real 1778628824] req@ffff95434701d500 x1865013828688384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628824.866194:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628824.866207:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628824.866218:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628824.866226:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828688384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628824.866237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628824.866247:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:3.0:1778628824.866248:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828688768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628824.866252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828688384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628824.866272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628824.866277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688512:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778628824.866287:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628824.866291:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628824.866301:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13159 00000800:00000100:0.0:1778628824.866306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628824.866307:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628824.866313:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628824.866320:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628824.866321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628824/real 1778628824] req@ffff95434701c380 x1865013828688512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628824.866345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828688512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628824.866346:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828688896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628824.866416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628824.866424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628824.866444:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628824.866451:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628824.866470:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628824.866477:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628824/real 1778628824] req@ffff95434701f100 x1865013828688640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628824.866485:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828688640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628824.866489:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628824.866491:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628824.866493:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628824.866497:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628824.866508:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628824.866513:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628824/real 1778628824] req@ffff95434701d880 x1865013828688768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628824.866520:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828688768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628824.866525:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628824.866527:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628824.866529:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628824.866540:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628824.866550:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628824.866561:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628824/real 1778628824] req@ffff95434701f800 x1865013828688896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628824.866569:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828688896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628824.866575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628824.866577:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628824.866579:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828688896:192.168.204.102@tcp:8:kworker.0 00010000:00010000:2.0:1778628825.057805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778628825.057828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628825.825780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628826.081833:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628826.081919:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628826.849773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628827.105799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628827.106000:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628827.873767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628828.129849:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628828.129991:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628828.897759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628829.153771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628829.153813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628829.921757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628829.985756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628829.985767:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628829.985772:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628829.985783:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13164 00000100:00080000:3.0:1778628829.985794:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628829.985800:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628829.985833:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828689024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628829.985860:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628829.985864:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628829.985880:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628829.985884:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13164 00000100:00080000:3.0:1778628829.985888:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628829.985892:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628829.985901:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628829.985915:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828689024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628829.985915:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828689152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628829.985930:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628829.985933:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628829.985934:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628829.985939:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13164 00000100:00080000:3.0:1778628829.985941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628829.985944:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628829.985948:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628829.985956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828689280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628829.985964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628829.985967:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628829.985968:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628829.985971:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13164 00000100:00080000:3.0:1778628829.985973:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628829.985976:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628829.985979:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628829.985989:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628829.985995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828689408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628829.986004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628829.986023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628829.986025:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628829.986029:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13164 00000100:00080000:3.0:1778628829.986035:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628829.986040:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628829.986046:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628829.986050:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628829/real 1778628829] req@ffff95434701c700 x1865013828689024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628829.986063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828689536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628829.986080:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828689024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628829.986093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628829.986099:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628829.986104:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828689024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628829.986128:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628829.986151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628829.986192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628829/real 1778628829] req@ffff95434701c000 x1865013828689152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628829.986207:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828689152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628829.986217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628829.986221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628829.986224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628829.986231:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628829.986270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628829.986282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628829/real 1778628829] req@ffff95434701ea00 x1865013828689280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628829.986305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828689280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628829.986321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628829.986327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628829.986333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628829.986345:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628829.986384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628829.986420:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628829/real 1778628829] req@ffff95434701d180 x1865013828689408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628829.986436:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828689408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628829.986453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628829.986456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628829.986460:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628829.986466:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628829.986489:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628829.986504:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628829/real 1778628829] req@ffff95434701ca80 x1865013828689536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628829.986523:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828689536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628829.986537:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628829.986542:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628829.986548:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628830.177779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628830.177878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628830.945757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628831.201804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628831.201937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628831.969805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628832.225882:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628832.225894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628832.993729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628833.249757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628833.249879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628834.017777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628834.273913:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628834.273926:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628835.041784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628835.105814:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628835.105824:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628835.105830:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628835.105842:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13169 00000100:00080000:1.0:1778628835.105856:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628835.105863:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628835.105908:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828689664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628835.105938:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628835.105941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628835.105942:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628835.105945:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13169 00000100:00080000:1.0:1778628835.105968:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628835.105972:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628835.105982:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628835.105989:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828689664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628835.106013:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828689792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628835.106023:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628835.106027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628835.106028:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628835.106032:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13169 00000100:00080000:1.0:1778628835.106034:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628835.106037:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628835.106041:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628835.106051:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828689920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628835.106077:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628835.106081:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628835.106083:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628835.106087:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13169 00000100:00080000:1.0:1778628835.106110:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628835.106116:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628835.106121:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628835.106154:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828690048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628835.106176:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689792:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778628835.106179:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628835.106183:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628835.106186:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628835.106190:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13169 00000100:00080000:1.0:1778628835.106195:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628835.106199:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628835.106205:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628835.106210:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628835.106223:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828690176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628835.106237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628835.106287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628835.297782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628835.297873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628836.065864:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628836.321755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628836.321834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628837.089862:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628837.345786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628837.345853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628838.113814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1778628838.177992:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:1.0:1778628838.178071:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1778628838.178093:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628838.178137:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628838.178148:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628838.178161:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:1.0:1778628838.178183:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778628838.178238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628835/real 1778628838] req@ffff954350194380 x1865013828690176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628838.178267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828690176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628838.178277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628838.178286:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628838.178290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628838.178295:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628835/real 1778628838] req@ffff954350194a80 x1865013828690048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628838.178303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828690048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628838.178308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628838.178310:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628838.178314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628838.178321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628835/real 1778628838] req@ffff954350194e00 x1865013828689792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628838.178333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828689792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628838.178367:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628838.178371:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628838.178375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628838.178383:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628835/real 1778628838] req@ffff954350196a00 x1865013828689920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628838.178395:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828689920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628838.178402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628838.178405:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628838.178408:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828689920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628838.178415:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628835/real 1778628838] req@ffff954350197b80 x1865013828689664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628838.178425:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828689664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628838.178432:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628838.178434:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628838.178437:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828689664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628838.369769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628838.369901:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628839.137775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628839.393736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628839.393781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628840.161801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628840.225819:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628840.225828:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628840.225833:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628840.225843:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13174 00000100:00080000:2.0:1778628840.225854:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628840.225861:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628840.225900:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828690304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628840.225931:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628840.225934:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628840.225935:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628840.225939:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13174 00000100:00080000:2.0:1778628840.225941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628840.225963:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628840.225974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628840.225984:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828690432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628840.225991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628840.225994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628840.225996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628840.225999:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13174 00000100:00080000:2.0:1778628840.226002:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628840.226005:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628840.226009:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628840.226015:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828690560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628840.226024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628840.226025:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828690304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628840.226031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628840.226035:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628840.226041:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13174 00000100:00080000:2.0:1778628840.226045:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628840.226049:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628840.226055:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628840.226068:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828690688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628840.226079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628840.226088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628840.226091:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628840.226095:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13174 00000100:00080000:2.0:1778628840.226100:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628840.226105:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628840.226112:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778628840.226112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628840.226129:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828690816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628840.226175:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628840/real 1778628840] req@ffff95435878a300 x1865013828690304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628840.226219:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828690304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628840.226231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628840.226237:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628840.226242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828690304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628840.226251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628840.226293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628840.226307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628840/real 1778628840] req@ffff95435878b800 x1865013828690432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628840.226321:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828690432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628840.226331:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628840.226334:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628840.226338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628840.226372:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628840.226396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628840.226408:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628840/real 1778628840] req@ffff954358789500 x1865013828690560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628840.226420:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828690560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628840.226462:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628840.226466:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628840.226470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628840.226477:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628840.226503:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628840.226515:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628840/real 1778628840] req@ffff954358789880 x1865013828690688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628840.226528:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828690688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628840.226536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628840.226539:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628840.226542:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628840.226548:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628840.226567:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628840.226577:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628840/real 1778628840] req@ffff954358788700 x1865013828690816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628840.226588:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828690816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628840.226598:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628840.226601:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628840.226612:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828690816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628840.417751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628840.417797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628841.185778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628841.441813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628841.441883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628842.209751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628842.465793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628842.465905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628843.233753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628843.489770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628843.489839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628844.257775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628844.513798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628844.513885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628845.281726:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628845.345752:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628845.345763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628845.345768:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628845.345779:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13179 00000100:00080000:2.0:1778628845.345791:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628845.345798:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628845.345843:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828690944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628845.345882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628845.345885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628845.345886:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628845.345890:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13179 00000100:00080000:2.0:1778628845.345892:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628845.345895:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628845.345903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628845.345913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828691072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628845.345940:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828690944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628845.345945:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628845.345948:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628845.345950:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628845.345953:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13179 00000100:00080000:2.0:1778628845.345956:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628845.345959:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628845.345963:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628845.345972:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828691200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628845.345979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628845.345982:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628845.345983:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628845.345986:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13179 00000100:00080000:2.0:1778628845.345988:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628845.345991:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628845.345993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628845.346000:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828691328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628845.346006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628845.346009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628845.346010:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628845.346012:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13179 00000100:00080000:2.0:1778628845.346019:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1778628845.346025:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628845.346025:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628845.346031:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628845.346040:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828691456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628845.346071:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628845/real 1778628845] req@ffff954358788e00 x1865013828690944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628845.346088:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828690944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628845.346098:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628845.346101:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628845.346104:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828690944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628845.346127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628845.346145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628845.346153:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628845/real 1778628845] req@ffff954358789c00 x1865013828691072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628845.346161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828691072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628845.346167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628845.346170:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628845.346179:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628845.346184:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628845.346198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628845.346204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628845/real 1778628845] req@ffff95435878b100 x1865013828691200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628845.346212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828691200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628845.346218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628845.346220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628845.346222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628845.346226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628845.346236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628845.346242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628845/real 1778628845] req@ffff95435878bb80 x1865013828691328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628845.346249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828691328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628845.346263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628845.346265:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628845.346267:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628845.346271:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628845.346282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628845.346295:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628845/real 1778628845] req@ffff95435878b480 x1865013828691456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628845.346303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828691456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628845.346309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628845.346311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628845.346313:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628845.537808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628845.537909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628846.305770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628846.561787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628846.561861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628847.329751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628847.585757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628847.585828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628848.353770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628848.609780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628848.609875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628849.377737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628849.633747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628849.633821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628850.401739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628850.465815:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628850.465826:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628850.465830:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628850.465841:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13184 00000100:00080000:1.0:1778628850.465853:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628850.465860:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628850.465905:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828691584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628850.465931:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628850.465934:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628850.465935:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628850.465938:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13184 00000100:00080000:1.0:1778628850.465941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628850.465944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628850.465953:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628850.465963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828691712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628850.465971:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628850.465975:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828691584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778628850.465975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628850.465995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628850.465999:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13184 00000100:00080000:1.0:1778628850.466003:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628850.466005:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628850.466010:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628850.466021:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828691840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628850.466028:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628850.466030:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628850.466031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628850.466034:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13184 00000100:00080000:1.0:1778628850.466045:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628850.466048:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628850.466051:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628850.466060:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628850.466072:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828691968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628850.466079:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628850.466082:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628850.466083:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628850.466086:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13184 00000100:00080000:1.0:1778628850.466088:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628850.466091:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628850.466094:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628850.466110:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628850/real 1778628850] req@ffff954350194700 x1865013828691584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628850.466111:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828692096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628850.466148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828691584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628850.466162:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628850.466167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628850.466172:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828691584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628850.466201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628850.466239:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628850.466260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628850/real 1778628850] req@ffff954350197800 x1865013828691712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628850.466270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828691712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628850.466277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628850.466279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628850.466281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628850.466286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628850.466301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628850.466311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628850/real 1778628850] req@ffff954350197480 x1865013828691840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628850.466324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828691840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628850.466333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628850.466336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628850.466340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628850.466346:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628850.466367:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628850.466378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628850/real 1778628850] req@ffff954350196d80 x1865013828691968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628850.466391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828691968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628850.466405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628850.466409:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628850.466422:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828691968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628850.466427:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628850.466444:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628850.466451:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628850/real 1778628850] req@ffff954350194000 x1865013828692096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628850.466459:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828692096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628850.466465:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628850.466467:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628850.466469:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628850.657751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628850.657826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628851.425741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628851.681762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628851.681941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628852.449744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628852.705743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628852.705813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628853.473752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628853.729786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628853.729878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628854.497748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628854.753768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628854.753854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628855.521785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628855.585810:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628855.585820:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628855.585825:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628855.585835:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13190 00000100:00080000:3.0:1778628855.585848:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628855.585855:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628855.585898:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828692224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628855.585927:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628855.585930:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628855.585932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628855.585939:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13190 00000100:00080000:3.0:1778628855.585942:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628855.585944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628855.585955:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628855.585965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828692352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628855.585973:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778628855.585978:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828692224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628855.585997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628855.585999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628855.586004:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13190 00000100:00080000:3.0:1778628855.586007:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628855.586010:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628855.586014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628855.586025:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828692480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628855.586032:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628855.586035:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628855.586036:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628855.586039:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13190 00000100:00080000:3.0:1778628855.586042:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628855.586044:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628855.586047:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:0.0:1778628855.586053:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628855.586059:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828692608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628855.586066:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628855.586069:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628855.586071:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628855.586074:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13190 00000100:00080000:3.0:1778628855.586076:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628855.586085:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628855.586094:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628855/real 1778628855] req@ffff95434701df80 x1865013828692224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778628855.586097:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628855.586112:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828692736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628855.586117:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828692224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628855.586148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628855.586152:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628855.586156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828692224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628855.586180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628855.586200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628855.586214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628855/real 1778628855] req@ffff95434701e680 x1865013828692352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628855.586227:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828692352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628855.586236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628855.586240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628855.586244:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628855.586251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628855.586271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628855.586279:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628855/real 1778628855] req@ffff95434701f480 x1865013828692480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628855.586288:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828692480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628855.586294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628855.586296:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628855.586298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628855.586309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628855.586326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628855.586332:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628855/real 1778628855] req@ffff95434701ca80 x1865013828692608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628855.586340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828692608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628855.586346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628855.586348:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628855.586350:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628855.586353:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628855.586363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628855.586368:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628855/real 1778628855] req@ffff95434701d180 x1865013828692736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628855.586375:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828692736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628855.586381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628855.586382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628855.586384:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628855.777769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628855.777869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628856.545796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628856.801766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628856.801851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628857.569764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628857.825770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628857.825862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628858.593756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628858.849747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628858.849828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628859.617803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628859.873778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628859.873849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628860.641844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628860.705825:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628860.705838:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628860.705844:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628860.705859:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13195 00000100:00080000:0.0:1778628860.705874:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628860.705881:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628860.705937:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828692864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628860.705971:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628860.705975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628860.705977:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628860.705982:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13195 00000100:00080000:0.0:1778628860.705986:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628860.705990:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628860.706006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628860.706033:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828692992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628860.706041:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828692864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628860.706044:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628860.706051:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628860.706054:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628860.706057:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13195 00000100:00080000:0.0:1778628860.706060:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628860.706063:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628860.706099:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628860.706111:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828693120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628860.706118:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628860.706120:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628860.706122:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628860.706124:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13195 00000100:00080000:0.0:1778628860.706130:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628860.706132:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628860.706136:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628860.706145:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828693248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628860.706147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628860.706155:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628860.706158:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628860.706159:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628860.706162:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13195 00000100:00080000:0.0:1778628860.706164:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628860.706167:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628860.706170:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628860.706176:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828693376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628860.706213:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628860/real 1778628860] req@ffff954342f4a680 x1865013828692864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628860.706245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828692864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628860.706258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628860.706263:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628860.706268:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828692864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628860.706297:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628860.706332:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628860.706346:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628860/real 1778628860] req@ffff954342f48a80 x1865013828692992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628860.706359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828692992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628860.706368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628860.706378:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628860.706381:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828692992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628860.706387:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628860.706417:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628860.706427:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628860/real 1778628860] req@ffff954342f4ad80 x1865013828693120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628860.706438:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828693120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628860.706445:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628860.706449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628860.706452:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628860.706458:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628860.706474:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628860.706483:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628860/real 1778628860] req@ffff954342f49500 x1865013828693248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628860.706494:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828693248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628860.706511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628860.706515:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628860.706518:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628860.706524:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628860.706549:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628860.706560:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628860/real 1778628860] req@ffff954342f49f80 x1865013828693376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628860.706572:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828693376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628860.706580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628860.706583:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628860.706586:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628860.897811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628860.897882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628861.665770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628861.921772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628861.921857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628862.689761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628862.945789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628862.945871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628863.713783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628863.969779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628863.969847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628864.737764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778628864.993773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628864.993830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628865.761782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628865.825814:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628865.825824:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628865.825829:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628865.825840:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13200 00000100:00080000:0.0:1778628865.825852:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628865.825859:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628865.825903:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828693504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628865.825948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628865.825967:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628865.825969:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628865.825973:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13200 00000100:00080000:0.0:1778628865.825976:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628865.825979:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628865.825989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628865.826002:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828693632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628865.826011:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628865.826013:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628865.826015:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628865.826018:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13200 00000100:00080000:0.0:1778628865.826021:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628865.826031:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828693504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628865.826060:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628865.826067:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628865.826083:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828693760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628865.826096:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628865.826100:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628865.826102:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628865.826107:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13200 00000100:00080000:0.0:1778628865.826116:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628865.826121:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628865.826127:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628865.826129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628865.826148:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828693888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628865.826170:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628865.826174:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628865.826177:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628865.826183:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13200 00000100:00080000:0.0:1778628865.826189:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628865.826194:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628865/real 1778628865] req@ffff954342f4b100 x1865013828693504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628865.826197:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628865.826207:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628865.826236:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828693504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628865.826239:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828694016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628865.826282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628865.826288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628865.826293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828693504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628865.826302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628865.826329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628865.826343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628865/real 1778628865] req@ffff954342f48700 x1865013828693632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628865.826370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828693632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628865.826379:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628865.826382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628865.826385:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628865.826412:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628865.826432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628865.826443:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628865/real 1778628865] req@ffff954342f49c00 x1865013828693760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628865.826456:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828693760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628865.826464:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628865.826467:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628865.826471:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628865.826477:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628865.826504:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628865.826514:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628865/real 1778628865] req@ffff954342f4b480 x1865013828693888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628865.826525:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828693888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628865.826546:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628865.826550:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628865.826553:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828693888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628865.826560:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628865.826581:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628865.826591:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628865/real 1778628865] req@ffff954342f4a300 x1865013828694016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628865.826601:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828694016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628865.826610:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628865.826613:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628865.826616:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778628866.017780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628866.017859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628866.785814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628867.042002:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628867.042047:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628867.809828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628868.066128:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778628868.066147:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628868.833767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628869.089887:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628869.089895:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628869.857774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778628870.114017:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628870.114126:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628870.881745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628870.945814:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628870.945849:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628870.945855:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628870.945868:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13205 00000100:00080000:0.0:1778628870.945884:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628870.945890:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628870.945935:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828694144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628870.945965:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628870.945968:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628870.945970:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628870.945973:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13205 00000100:00080000:0.0:1778628870.945976:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628870.945978:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628870.945988:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628870.945999:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828694272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628870.946007:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628870.946011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628870.946029:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628870.946036:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13205 00000100:00100000:1.0:1778628870.946037:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828694144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628870.946041:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628870.946048:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628870.946057:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628870.946067:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828694400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628870.946076:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628870.946079:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628870.946080:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628870.946085:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13205 00000100:00080000:0.0:1778628870.946088:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628870.946091:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628870.946095:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628870.946105:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828694528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628870.946111:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628870.946113:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628870.946115:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628870.946117:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13205 00000100:00080000:0.0:1778628870.946123:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628870.946125:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628870.946130:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:1.0:1778628870.946134:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628870.946156:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828694656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628870.946186:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628870/real 1778628870] req@ffff954342f49880 x1865013828694144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628870.946212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828694144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628870.946220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628870.946224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628870.946247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828694144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628870.946254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628870.946274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628870.946283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628870/real 1778628870] req@ffff954342f4bb80 x1865013828694272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628870.946292:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828694272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628870.946298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628870.946300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628870.946302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628870.946322:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628870.946333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628870.946341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628870/real 1778628870] req@ffff954342f48e00 x1865013828694400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628870.946353:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828694400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628870.946362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628870.946365:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628870.946369:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628870.946376:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628870.946397:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628870.946411:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628870/real 1778628870] req@ffff954342f4aa00 x1865013828694528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628870.946419:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828694528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628870.946425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628870.946427:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628870.946429:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628870.946434:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628870.946446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628870.946452:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628870/real 1778628870] req@ffff954342f49180 x1865013828694656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628870.946467:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828694656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628870.946473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628870.946476:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628870.946478:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628871.137802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628871.137860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628871.905771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628872.161888:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628872.161982:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628872.929804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628873.185767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628873.185819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628873.953794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628874.209786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628874.209865:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628874.977788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628875.233788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628875.233856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628876.001814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628876.065799:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628876.065807:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628876.065811:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628876.065819:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13210 00000100:00080000:1.0:1778628876.065846:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628876.065852:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628876.065914:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828694784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628876.065939:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628876.065941:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628876.065943:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628876.065946:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13210 00000100:00080000:1.0:1778628876.065950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628876.065953:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628876.065962:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628876.065995:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828694784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628876.066001:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828694912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628876.066009:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628876.066011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628876.066013:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628876.066015:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13210 00000100:00080000:1.0:1778628876.066018:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628876.066020:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628876.066024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628876.066038:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828695040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628876.066044:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628876.066047:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628876.066048:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628876.066058:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13210 00000800:00000100:0.0:1778628876.066061:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778628876.066063:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628876.066067:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628876.066072:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628876.066081:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828695168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628876.066088:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628876.066091:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628876.066092:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628876.066095:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13210 00000100:00080000:1.0:1778628876.066097:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628876.066100:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628876.066103:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628876.066105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628876/real 1778628876] req@ffff954350197100 x1865013828694784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628876.066110:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828695296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628876.066140:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828694784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628876.066148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628876.066152:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628876.066155:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828694784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628876.066173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628876.066190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628876.066201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628876/real 1778628876] req@ffff954350194000 x1865013828694912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628876.066233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828694912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628876.066243:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628876.066246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628876.066250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828694912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628876.066257:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628876.066274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628876.066283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628876/real 1778628876] req@ffff954350196d80 x1865013828695040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628876.066291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828695040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628876.066296:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628876.066298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628876.066300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628876.066304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628876.066314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628876.066319:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628876/real 1778628876] req@ffff954350197480 x1865013828695168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628876.066326:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828695168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628876.066348:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628876.066350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628876.066352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628876.066356:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628876.066370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628876.066376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628876/real 1778628876] req@ffff954350197800 x1865013828695296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628876.066383:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828695296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628876.066389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628876.066391:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628876.066393:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695296:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778628876.257780:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628876.257814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778628877.025770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628877.281803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628877.281830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628878.049806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628878.305750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628878.305786:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628879.073782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628879.329751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628879.329811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628880.097790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628880.353746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628880.353790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628881.121815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778628881.185773:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628881.185784:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778628881.185790:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628881.185801:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13215 00000100:00080000:1.0:1778628881.185816:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628881.185823:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628881.185873:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828695424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628881.185906:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628881.185933:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778628881.185936:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628881.185941:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13215 00000100:00080000:1.0:1778628881.185946:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628881.185951:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628881.185962:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628881.185975:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828695424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628881.185978:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828695552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628881.185991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628881.185997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778628881.185999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628881.186005:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13215 00000100:00080000:1.0:1778628881.186013:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628881.186018:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628881.186026:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628881.186044:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828695680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628881.186058:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628881.186063:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778628881.186065:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628881.186071:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13215 00000100:00080000:1.0:1778628881.186077:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628881.186082:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:0.0:1778628881.186091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778628881.186102:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628881.186120:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828695808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628881.186129:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628881.186132:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778628881.186134:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778628881.186138:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13215 00000100:00080000:1.0:1778628881.186144:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628881.186148:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778628881.186153:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628881.186156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628881/real 1778628881] req@ffff954350194700 x1865013828695424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628881.186164:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828695936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778628881.186213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828695424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628881.186229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628881.186234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628881.186238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828695424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628881.186268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628881.186292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628881.186304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628881/real 1778628881] req@ffff954350197b80 x1865013828695552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628881.186315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828695552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628881.186333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628881.186336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628881.186339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628881.186343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778628881.186356:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628881.186363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628881/real 1778628881] req@ffff954350196a00 x1865013828695680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628881.186387:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828695680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628881.186395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628881.186397:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628881.186400:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628881.186405:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628881.186422:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628881.186430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628881/real 1778628881] req@ffff954350194e00 x1865013828695808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628881.186439:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828695808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628881.186446:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628881.186449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628881.186451:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628881.186455:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778628881.186468:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778628881.186478:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628881/real 1778628881] req@ffff954350194a80 x1865013828695936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628881.186495:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828695936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628881.186534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778628881.186539:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778628881.186544:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828695936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628881.377733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628881.377795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628882.145805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628882.401775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628882.401828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628883.169804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628883.425763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628883.425790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628884.193755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628884.449766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628884.449791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628885.217799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628885.473780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628885.473811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628886.241807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628886.305761:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628886.305772:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628886.305777:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628886.305787:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13220 00000100:00080000:0.0:1778628886.305797:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628886.305804:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628886.305848:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828696064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628886.305876:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628886.305880:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628886.305882:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628886.305886:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13220 00000100:00080000:0.0:1778628886.305902:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628886.305907:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628886.305914:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828696064:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778628886.305917:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628886.305933:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828696192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628886.305945:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628886.305949:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628886.305951:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628886.305955:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13220 00000100:00080000:0.0:1778628886.305967:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628886.305971:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778628886.305977:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778628886.305978:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628886.305993:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828696320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628886.306004:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628886.306008:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628886/real 1778628886] req@ffff954342f48000 x1865013828696064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628886.306009:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628886.306013:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628886.306024:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13220 00000100:00080000:0.0:1778628886.306030:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628886.306032:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828696064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628886.306037:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:1.0:1778628886.306043:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778628886.306045:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:1.0:1778628886.306049:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628886.306053:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828696064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628886.306063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828696448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628886.306072:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696192:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778628886.306096:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628886.306100:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628886.306103:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778628886.306104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628886.306112:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13220 00000100:00080000:0.0:1778628886.306118:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628886.306118:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628886/real 1778628886] req@ffff954342f48380 x1865013828696192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628886.306127:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628886.306132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828696192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778628886.306134:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628886.306143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628886.306146:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628886.306149:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778628886.306154:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828696576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628886.306157:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628886.306177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628886.306185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628886/real 1778628886] req@ffff954342f49180 x1865013828696320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628886.306195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828696320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628886.306200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628886.306202:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628886.306204:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628886.306214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628886.306225:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628886.306230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628886/real 1778628886] req@ffff954342f4aa00 x1865013828696448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628886.306247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828696448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628886.306252:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628886.306254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628886.306256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628886.306259:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628886.306271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628886.306276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628886/real 1778628886] req@ffff954342f48e00 x1865013828696576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628886.306283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828696576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628886.306289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628886.306291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628886.306293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628886.497781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628886.497806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628887.265750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628887.521746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628887.521772:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628888.289743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628888.545799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628888.545837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628889.313750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628889.569818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628889.569870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628890.337781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628890.593798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628890.593886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628891.361754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628891.425805:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628891.425818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628891.425835:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628891.425847:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13225 00000100:00080000:3.0:1778628891.425862:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628891.425871:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628891.425942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828696704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628891.425973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628891.425977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628891.425979:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628891.425984:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13225 00000100:00080000:3.0:1778628891.425989:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628891.426016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828696704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628891.426016:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628891.426029:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628891.426047:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828696832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628891.426060:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628891.426064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628891.426066:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628891.426070:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13225 00000100:00080000:3.0:1778628891.426076:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628891.426080:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628891.426086:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778628891.426091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628891.426100:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828696960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628891.426112:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628891.426115:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628891.426118:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628891.426123:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13225 00000100:00080000:3.0:1778628891.426127:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628891.426131:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628891/real 1778628891] req@ffff95434701ea00 x1865013828696704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628891.426136:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628891.426146:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628891.426156:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828696704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628891.426161:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828697088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628891.426188:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778628891.426192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628891.426197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628891.426203:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828696704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628891.426204:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628891.426208:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628891.426215:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13225 00000100:00080000:3.0:1778628891.426230:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628891.426235:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628891.426240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696832:192.168.204.102@tcp:38:kworker.0 00000008:00100000:3.0:1778628891.426241:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628891.426261:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828697216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628891.426268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628891.426282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628891/real 1778628891] req@ffff95434701c000 x1865013828696832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628891.426291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828696832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628891.426298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628891.426302:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628891.426304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628891.426309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628891.426320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628891.426325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628891/real 1778628891] req@ffff95434701c700 x1865013828696960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628891.426333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828696960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628891.426338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628891.426340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628891.426341:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828696960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628891.426351:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628891.426366:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628891.426372:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628891/real 1778628891] req@ffff95434701f800 x1865013828697088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628891.426390:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828697088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628891.426396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628891.426398:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628891.426400:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628891.426425:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628891.426443:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628891.426450:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628891/real 1778628891] req@ffff95434701d880 x1865013828697216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628891.426469:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828697216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628891.426475:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628891.426482:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628891.426484:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628891.617798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628891.617900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628892.385761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628892.641750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628892.641868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628893.409732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628893.665776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628893.665837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628894.433777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628894.689730:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628894.689819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628895.457828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628895.713739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628895.713833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628896.481783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628896.545780:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628896.545793:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628896.545798:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628896.545810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13231 00000100:00080000:3.0:1778628896.545824:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628896.545833:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628896.545884:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828697344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628896.545920:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628896.545924:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628896.545927:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628896.545931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13231 00000100:00080000:3.0:1778628896.545936:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628896.545941:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628896.545952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628896.545965:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828697472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628896.545980:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828697344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628896.545997:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628896.545999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628896.546001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628896.546004:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13231 00000100:00080000:3.0:1778628896.546008:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628896.546011:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628896.546015:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628896.546026:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828697600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628896.546033:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628896.546035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628896.546037:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628896.546039:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13231 00000100:00080000:3.0:1778628896.546041:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628896.546044:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628896.546047:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628896.546055:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828697728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628896.546061:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628896.546063:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628896.546064:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778628896.546070:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628896.546070:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13231 00000100:00080000:3.0:1778628896.546078:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628896.546082:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628896.546086:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628896.546095:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828697856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628896.546120:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628896/real 1778628896] req@ffff95434701f100 x1865013828697344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628896.546141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828697344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628896.546155:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628896.546159:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628896.546164:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828697344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628896.546189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628896.546213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628896.546227:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628896/real 1778628896] req@ffff95434701c380 x1865013828697472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628896.546241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828697472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628896.546250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628896.546254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628896.546257:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628896.546264:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628896.546285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628896.546297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628896/real 1778628896] req@ffff95434701d500 x1865013828697600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628896.546337:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828697600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628896.546346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628896.546349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628896.546353:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628896.546360:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628896.546384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628896.546395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628896/real 1778628896] req@ffff95434701d880 x1865013828697728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628896.546408:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828697728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628896.546418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628896.546422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628896.546425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628896.546432:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628896.546449:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628896.546459:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628896/real 1778628896] req@ffff95434701f800 x1865013828697856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628896.546472:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828697856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628896.546494:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628896.546499:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628896.546502:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828697856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628896.737741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628896.737815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628897.505795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628897.761769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628897.761795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628898.529781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628898.785787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628898.785861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628899.553779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628899.809769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628899.809848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628900.577760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628900.833788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628900.833853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628901.601801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628901.665785:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628901.665795:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628901.665800:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628901.665810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13236 00000100:00080000:2.0:1778628901.665844:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628901.665852:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628901.665902:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828697984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628901.665931:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628901.665935:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628901.665936:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628901.665940:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13236 00000100:00080000:2.0:1778628901.665942:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628901.665944:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628901.665954:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628901.665969:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828698112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628901.665980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628901.665984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628901.665986:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628901.665996:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828697984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628901.666000:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13236 00000100:00080000:2.0:1778628901.666007:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628901.666012:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628901.666017:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628901.666030:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828698240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628901.666038:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628901.666048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628901.666050:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628901.666053:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13236 00000100:00080000:2.0:1778628901.666057:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628901.666061:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628901.666065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628901.666082:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828698368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628901.666090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628901.666093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628901.666094:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628901.666096:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13236 00000100:00080000:2.0:1778628901.666098:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628901.666101:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628901.666106:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628901.666114:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828698496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628901.666120:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628901.666164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628901.666181:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628901.666204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628901.857755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628901.857850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628902.625776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628902.881769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628902.881816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628903.649812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628903.905773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628903.905840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628904.673800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778628904.737938:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778628904.737979:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778628904.737996:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628904.738046:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628904.738057:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628904.738065:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778628904.738076:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778628904.738103:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628901/real 1778628904] req@ffff95435878b480 x1865013828698240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628904.738133:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828698240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628904.738145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628904.738180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628904.738185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628904.738193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628901/real 1778628904] req@ffff95435878bb80 x1865013828698368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628904.738208:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828698368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628904.738217:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628904.738221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628904.738224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628904.738231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628901/real 1778628904] req@ffff95435878b100 x1865013828698496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628904.738242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828698496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628904.738250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628904.738254:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628904.738257:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628904.738264:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628901/real 1778628904] req@ffff95435878aa00 x1865013828697984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628904.738276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828697984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628904.738283:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628904.738295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628904.738299:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828697984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628904.738306:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628901/real 1778628904] req@ffff954358788380 x1865013828698112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628904.738318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828698112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628904.738326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628904.738330:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628904.738333:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778628904.929800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628904.929877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628905.697732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628905.953752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628905.953844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628906.721750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628906.785813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628906.785823:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628906.785827:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628906.785837:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13241 00000100:00080000:2.0:1778628906.785849:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628906.785855:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628906.785907:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828698624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628906.785944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628906.785950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628906.785952:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628906.785958:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13241 00000100:00080000:2.0:1778628906.785963:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628906.785969:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628906.785980:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628906.785997:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828698624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628906.786000:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828698752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628906.786032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628906.786035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628906.786037:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628906.786041:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13241 00000100:00080000:2.0:1778628906.786045:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628906.786048:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628906.786070:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628906.786085:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828698880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628906.786095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628906.786098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628906.786101:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778628906.786103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628906.786123:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13241 00000100:00080000:2.0:1778628906.786130:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628906.786135:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628906.786141:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628906.786155:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828699008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628906.786161:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628906/real 1778628906] req@ffff954358789c00 x1865013828698624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628906.786174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628906.786178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628906.786180:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628906.786185:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13241 00000100:00080000:2.0:1778628906.786194:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628906.786198:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628906.786203:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628906.786205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828698624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628906.786217:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828699136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628906.786219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628906.786229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628906.786236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828698624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628906.786274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628906.786301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628906.786324:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628906/real 1778628906] req@ffff954358788e00 x1865013828698752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628906.786336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828698752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628906.786345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628906.786349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628906.786352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628906.786358:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628906.786384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628906.786395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628906/real 1778628906] req@ffff954358788700 x1865013828698880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628906.786413:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828698880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628906.786420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628906.786424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628906.786427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828698880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628906.786434:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628906.786455:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628906.786466:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628906/real 1778628906] req@ffff954358789880 x1865013828699008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628906.786477:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828699008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628906.786493:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628906.786496:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628906.786499:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628906.786504:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628906.786525:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628906.786534:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628906/real 1778628906] req@ffff954358789500 x1865013828699136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628906.786544:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828699136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628906.786552:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628906.786556:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628906.786558:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628906.977771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628906.977873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628907.745746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628908.001768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628908.001893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628908.769773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628909.025745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628909.025852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628909.793741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628910.049757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628910.049822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628910.817769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628911.073762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628911.073869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628911.841736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628911.905810:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628911.905821:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628911.905826:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628911.905838:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13246 00000100:00080000:0.0:1778628911.905850:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628911.905857:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628911.905919:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828699264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628911.905951:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628911.905954:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628911.905956:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628911.905959:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13246 00000100:00080000:0.0:1778628911.905962:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628911.905964:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628911.905973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628911.905984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828699392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628911.905991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628911.905994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628911.905996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628911.906001:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13246 00000100:00080000:0.0:1778628911.906004:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628911.906022:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628911.906029:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628911.906034:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828699264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628911.906047:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828699520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628911.906060:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628911.906064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628911.906066:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628911.906079:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13246 00000100:00080000:0.0:1778628911.906086:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628911.906091:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628911.906097:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628911.906110:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828699648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628911.906124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628911.906135:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628911.906139:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628911.906142:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628911.906148:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13246 00000100:00080000:0.0:1778628911.906152:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628911.906157:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628911.906163:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628911.906178:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828699776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628911.906178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628911/real 1778628911] req@ffff954342f4bb80 x1865013828699264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628911.906238:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828699264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628911.906268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628911.906273:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628911.906278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828699264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628911.906286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628911.906308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628911.906318:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628911/real 1778628911] req@ffff954342f49880 x1865013828699392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628911.906327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828699392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628911.906333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628911.906336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628911.906338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628911.906358:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628911.906370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628911.906376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628911/real 1778628911] req@ffff954342f4a300 x1865013828699520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628911.906383:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828699520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628911.906388:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628911.906390:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628911.906391:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628911.906402:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628911.906415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628911.906422:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628911/real 1778628911] req@ffff954342f4b480 x1865013828699648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628911.906429:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828699648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628911.906434:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628911.906436:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628911.906438:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628911.906442:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628911.906452:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628911.906457:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628911/real 1778628911] req@ffff954342f49c00 x1865013828699776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628911.906464:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828699776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628911.906469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628911.906471:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628911.906473:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828699776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628912.097744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628912.097888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628912.865748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628913.121772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628913.121840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628913.889800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628914.145783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628914.145913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628914.913767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628915.169870:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628915.169903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628915.937782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628916.193766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628916.193859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628916.961778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628917.025809:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628917.025817:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628917.025821:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628917.025830:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13251 00000100:00080000:3.0:1778628917.025840:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628917.025846:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628917.025884:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828699904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628917.025917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628917.025920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628917.025922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628917.025925:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13251 00000100:00080000:3.0:1778628917.025941:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628917.025944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628917.025954:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628917.025964:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828700032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628917.025972:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628917.025975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628917.025976:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628917.025979:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13251 00000100:00080000:3.0:1778628917.025981:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628917.025983:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628917.025988:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628917.025997:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828700160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628917.026005:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628917.026009:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1778628917.026010:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828699904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628917.026011:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628917.026024:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13251 00000100:00080000:3.0:1778628917.026027:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628917.026029:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628917.026034:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628917.026045:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828700288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628917.026061:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628917.026064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628917.026066:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628917.026069:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13251 00000100:00080000:3.0:1778628917.026072:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628917.026079:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628917.026084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628917.026093:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828700416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628917.026103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628917.026150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628917/real 1778628917] req@ffff95434701c700 x1865013828699904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628917.026167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828699904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628917.026177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628917.026180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628917.026184:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828699904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628917.026190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628917.026205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628917.026212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628917/real 1778628917] req@ffff95434701c000 x1865013828700032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628917.026220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828700032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628917.026226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628917.026228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628917.026230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628917.026252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628917.026262:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628917.026268:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628917/real 1778628917] req@ffff95434701ea00 x1865013828700160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628917.026289:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828700160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628917.026295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628917.026297:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628917.026299:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628917.026303:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628917.026316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628917.026322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628917/real 1778628917] req@ffff95434701d180 x1865013828700288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628917.026329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828700288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628917.026334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628917.026336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628917.026338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628917.026341:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628917.026350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628917.026356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628917/real 1778628917] req@ffff95434701ca80 x1865013828700416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628917.026363:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828700416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628917.026377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628917.026379:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628917.026382:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628917.217783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628917.217797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628917.985807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628918.241788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628918.241846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628919.009743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628919.265756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628919.265824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628920.033831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628920.289786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628920.289890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628921.057754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628921.313755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628921.313848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628922.081835:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628922.145733:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628922.145743:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628922.145748:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628922.145757:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13256 00000100:00080000:3.0:1778628922.145766:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628922.145771:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628922.145801:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828700544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628922.145844:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628922.145848:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628922.145850:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628922.145854:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13256 00000100:00080000:3.0:1778628922.145857:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628922.145859:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628922.145879:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628922.145892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828700544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628922.145892:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828700672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628922.145907:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628922.145910:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628922.145911:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628922.145915:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13256 00000100:00080000:3.0:1778628922.145919:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628922.145923:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628922.145929:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628922.145941:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828700800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628922.145952:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778628922.145956:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628922.145960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628922.145962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628922.145968:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13256 00000100:00080000:3.0:1778628922.145972:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628922.145977:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628922.145981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628922.145990:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628922/real 1778628922] req@ffff95434701f480 x1865013828700544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628922.145994:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828700928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628922.146015:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628922.146018:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778628922.146022:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828700544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628922.146027:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628922.146037:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13256 00000100:00080000:1.0:1778628922.146038:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628922.146045:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778628922.146045:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628922.146049:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828700544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628922.146054:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628922.146061:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628922.146069:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778628922.146078:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828701056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628922.146100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628922.146111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628922/real 1778628922] req@ffff95434701e680 x1865013828700672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628922.146126:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828700672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628922.146139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628922.146142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628922.146145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628922.146149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628922.146163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628922.146169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628922/real 1778628922] req@ffff95434701df80 x1865013828700800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628922.146176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828700800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628922.146182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628922.146184:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628922.146185:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628922.146189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628922.146198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628922.146203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628922/real 1778628922] req@ffff95434701ca80 x1865013828700928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628922.146210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828700928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628922.146215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628922.146217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628922.146229:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828700928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628922.146241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628922.146254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628922.146261:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628922/real 1778628922] req@ffff95434701d180 x1865013828701056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628922.146268:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828701056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628922.146274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628922.146276:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628922.146278:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628922.337776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628922.337847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628923.105829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628923.361777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628923.361870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628924.129784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628924.385753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628924.385814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628925.153811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628925.409769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628925.409818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628926.177814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628926.433803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628926.433842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628927.201747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628927.265805:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628927.265812:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628927.265817:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628927.265824:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13261 00000100:00080000:2.0:1778628927.265834:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628927.265839:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628927.265870:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828701184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628927.265891:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628927.265894:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628927.265895:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628927.265902:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13261 00000100:00080000:2.0:1778628927.265904:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628927.265907:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628927.265915:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628927.265924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828701312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628927.265932:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628927.265934:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628927.265936:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628927.265943:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828701184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628927.265953:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13261 00000100:00080000:2.0:1778628927.265957:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628927.265960:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628927.265963:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628927.265972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828701440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628927.265980:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628927.265983:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628927.265984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628927.265986:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13261 00000100:00080000:2.0:1778628927.265988:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628927.265990:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628927.265994:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628927.266001:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828701568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628927.266006:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628927.266020:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628927.266023:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628927.266024:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628927.266028:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13261 00000100:00080000:2.0:1778628927.266031:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628927.266034:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628927.266038:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628927.266044:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628927/real 1778628927] req@ffff95435878b800 x1865013828701184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628927.266046:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828701696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628927.266085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828701184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628927.266097:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628927.266102:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628927.266115:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828701184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628927.266123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628927.266142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628927.266151:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628927/real 1778628927] req@ffff95435878a300 x1865013828701312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628927.266160:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828701312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628927.266166:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628927.266170:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628927.266174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628927.266198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628927.266220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628927.266231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628927/real 1778628927] req@ffff95435878ad80 x1865013828701440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628927.266244:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828701440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628927.266253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628927.266256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628927.266260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628927.266266:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628927.266285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628927.266299:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628927/real 1778628927] req@ffff954358789180 x1865013828701568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628927.266308:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828701568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628927.266314:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628927.266316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628927.266318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628927.266322:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628927.266333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628927.266338:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628927/real 1778628927] req@ffff954358788a80 x1865013828701696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628927.266345:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828701696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628927.266351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628927.266353:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628927.266354:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628927.457771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628927.457835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628928.225785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628928.481808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628928.481900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628929.249784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628929.505789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628929.505881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628930.273774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628930.529770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628930.529829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628931.297772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628931.553765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628931.553833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628932.321757:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628932.385770:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628932.385778:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628932.385782:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628932.385790:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13266 00000100:00080000:2.0:1778628932.385801:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628932.385809:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628932.385849:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828701824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628932.385875:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628932.385877:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628932.385883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628932.385886:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13266 00000100:00080000:2.0:1778628932.385889:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628932.385892:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628932.385901:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628932.385921:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828701824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628932.385940:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828701952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628932.385949:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628932.385951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628932.385952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628932.385955:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13266 00000100:00080000:2.0:1778628932.385959:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628932.385961:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628932.385964:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628932.385972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828702080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628932.385980:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628932.385981:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628932.385984:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628932.385985:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628932.385989:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13266 00000100:00080000:2.0:1778628932.385991:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628932.385994:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628932.385998:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628932.386009:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828702208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628932.386019:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628932.386020:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628932/real 1778628932] req@ffff954358789f80 x1865013828701824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628932.386026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628932.386030:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778628932.386041:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828701824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778628932.386044:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13266 00000100:00080000:2.0:1778628932.386053:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628932.386055:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778628932.386060:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:1.0:1778628932.386061:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628932.386066:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828701824:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778628932.386067:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628932.386081:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828702336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628932.386090:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628932.386109:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628932.386117:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628932/real 1778628932] req@ffff95435878a680 x1865013828701952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628932.386125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828701952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628932.386131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628932.386133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628932.386135:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828701952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628932.386139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628932.386148:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628932.386153:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628932/real 1778628932] req@ffff954358788a80 x1865013828702080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628932.386160:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828702080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628932.386174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628932.386176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628932.386178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628932.386182:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628932.386195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628932.386200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628932/real 1778628932] req@ffff954358789180 x1865013828702208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628932.386208:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828702208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628932.386213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628932.386215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628932.386216:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628932.386226:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628932.386236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628932.386241:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628932/real 1778628932] req@ffff95435878ad80 x1865013828702336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628932.386248:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828702336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628932.386253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628932.386255:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628932.386261:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628932.577777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628932.577841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628933.345771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628933.601886:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628933.602125:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628934.369850:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628934.625930:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628934.625963:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628935.393782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628935.649918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628935.650067:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628936.417736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1778628936.673833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628936.674168:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628937.441790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628937.505818:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628937.505838:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628937.505845:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628937.505858:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13272 00000100:00080000:3.0:1778628937.505873:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628937.505881:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628937.505929:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828702464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628937.505968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628937.505972:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628937.505974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628937.505981:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13272 00000100:00080000:3.0:1778628937.505985:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628937.505991:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628937.506003:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628937.506020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828702592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628937.506030:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828702464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628937.506060:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628937.506064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628937.506067:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628937.506073:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13272 00000100:00080000:3.0:1778628937.506078:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628937.506083:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628937.506089:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628937.506103:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828702720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778628937.506114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628937.506116:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628937.506121:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628937.506123:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628937.506129:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13272 00000100:00080000:3.0:1778628937.506134:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628937.506139:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628937.506146:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628937.506160:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828702848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628937.506169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628937/real 1778628937] req@ffff95434701ea00 x1865013828702464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628937.506173:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628937.506183:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628937.506188:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628937.506196:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13272 00000100:00100000:1.0:1778628937.506202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828702464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628937.506211:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778628937.506219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778628937.506219:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:1.0:1778628937.506228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:3.0:1778628937.506229:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628937.506234:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828702464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628937.506249:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828702976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628937.506267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628937.506293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628937.506321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628937/real 1778628937] req@ffff95434701c000 x1865013828702592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628937.506333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828702592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628937.506342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628937.506346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628937.506356:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628937.506363:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628937.506384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628937.506395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628937/real 1778628937] req@ffff95434701c700 x1865013828702720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628937.506406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828702720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628937.506414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628937.506417:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628937.506420:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628937.506427:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628937.506446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628937.506455:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628937/real 1778628937] req@ffff95434701f800 x1865013828702848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628937.506466:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828702848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628937.506474:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628937.506477:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628937.506480:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628937.506486:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628937.506515:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628937.506534:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628937/real 1778628937] req@ffff95434701d880 x1865013828702976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628937.506546:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828702976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628937.506555:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628937.506558:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628937.506561:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828702976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628937.697877:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628937.698017:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628938.465841:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628938.721937:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628938.722259:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628939.489831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628939.745739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628939.745781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628940.513750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628940.769768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628940.769833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628941.537771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628941.793793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628941.793861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628942.561770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628942.625787:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628942.625799:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628942.625805:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628942.625816:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13277 00000100:00080000:2.0:1778628942.625830:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628942.625858:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628942.625897:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828703104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628942.625931:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628942.625935:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628942.625937:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628942.625941:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13277 00000100:00080000:2.0:1778628942.625946:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628942.625949:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628942.625960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628942.625974:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828703232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628942.625985:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828703104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628942.625987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628942.625993:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628942.625996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628942.626001:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13277 00000100:00080000:2.0:1778628942.626004:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628942.626007:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628942.626012:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628942.626023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828703360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628942.626032:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628942.626036:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628942.626042:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628942.626047:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13277 00000100:00080000:2.0:1778628942.626052:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628942.626057:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628942.626062:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628942.626067:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778628942.626076:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828703488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628942.626088:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628942.626092:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628942.626094:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628942.626098:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13277 00000100:00080000:2.0:1778628942.626101:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628942.626105:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628942.626110:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628942.626112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628942/real 1778628942] req@ffff95435878a300 x1865013828703104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778628942.626124:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828703616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628942.626136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828703104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628942.626149:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628942.626154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628942.626158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828703104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628942.626183:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628942.626210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628942.626221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628942/real 1778628942] req@ffff95435878b800 x1865013828703232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628942.626232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828703232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628942.626240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628942.626243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628942.626246:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628942.626252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628942.626266:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628942.626273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628942/real 1778628942] req@ffff954358789500 x1865013828703360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628942.626284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828703360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628942.626291:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628942.626294:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628942.626297:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628942.626302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628942.626318:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628942.626325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628942/real 1778628942] req@ffff954358789880 x1865013828703488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628942.626347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828703488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628942.626355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628942.626358:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628942.626360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628942.626365:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628942.626380:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628942.626388:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628942/real 1778628942] req@ffff954358788700 x1865013828703616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628942.626398:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828703616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628942.626422:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628942.626425:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628942.626428:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628942.817771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628942.817846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628943.585782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628943.841810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628943.841870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628944.609805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628944.865798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628944.865884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628945.633798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628945.889806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628945.889889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628946.657753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628946.913802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628946.913880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628947.681754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628947.745808:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628947.745819:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628947.745824:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628947.745834:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13282 00000100:00080000:3.0:1778628947.745847:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628947.745853:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628947.745895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828703744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628947.745922:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628947.745925:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628947.745926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628947.745930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13282 00000100:00080000:3.0:1778628947.745932:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628947.745934:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628947.745943:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628947.745954:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828703872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628947.745963:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628947.745974:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828703744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628947.745986:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628947.745989:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628947.745994:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13282 00000100:00080000:3.0:1778628947.745999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628947.746004:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628947.746010:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628947.746026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828704000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628947.746037:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628947.746041:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1778628947.746045:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628947.746045:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628947.746054:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13282 00000100:00080000:3.0:1778628947.746060:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628947.746064:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628947.746071:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628947.746085:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828704128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628947.746091:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628947/real 1778628947] req@ffff95434701d500 x1865013828703744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628947.746095:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628947.746099:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628947.746102:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628947.746110:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13282 00000100:00080000:3.0:1778628947.746118:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628947.746122:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628947.746125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828703744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628947.746138:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1778628947.746140:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778628947.746144:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628947.746148:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828703744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628947.746161:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828704256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628947.746182:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628947.746210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628947.746225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628947/real 1778628947] req@ffff95434701c380 x1865013828703872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628947.746239:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828703872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628947.746249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628947.746253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628947.746257:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828703872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628947.746263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628947.746283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628947.746293:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628947/real 1778628947] req@ffff95434701f100 x1865013828704000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628947.746323:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828704000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628947.746345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628947.746350:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628947.746354:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628947.746361:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628947.746385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628947.746396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628947/real 1778628947] req@ffff95434701d880 x1865013828704128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628947.746409:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828704128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628947.746419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628947.746422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628947.746426:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628947.746433:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628947.746451:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628947.746461:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628947/real 1778628947] req@ffff95434701f800 x1865013828704256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628947.746474:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828704256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628947.746483:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628947.746486:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628947.746496:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628947.937793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628947.937852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628948.705779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628948.961780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628948.961861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628949.729791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628949.985804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628949.985871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628950.753741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628951.009778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628951.009867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628951.777805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628952.033787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628952.033817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628952.801787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778628952.865777:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628952.865786:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778628952.865790:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628952.865798:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13287 00000100:00080000:2.0:1778628952.865807:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628952.865813:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778628952.865849:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828704384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628952.865882:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628952.865886:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778628952.865888:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628952.865892:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13287 00000100:00080000:2.0:1778628952.865896:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628952.865900:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628952.865907:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628952.865921:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828704512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628952.865936:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828704384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778628952.865960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628952.865962:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778628952.865964:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628952.865968:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13287 00000100:00080000:2.0:1778628952.865971:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628952.865974:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628952.865979:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778628952.865988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828704640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628952.865995:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628952.865998:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778628952.865999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628952.866002:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13287 00000100:00080000:2.0:1778628952.866007:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628952.866009:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628952.866012:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778628952.866018:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828704768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778628952.866025:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778628952.866027:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778628952.866028:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778628952.866030:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13287 00000800:00000100:1.0:1778628952.866035:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778628952.866039:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778628952.866044:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778628952.866049:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778628952.866067:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828704896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628952.866080:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628952/real 1778628952] req@ffff954358788e00 x1865013828704384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628952.866107:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828704384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628952.866117:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628952.866120:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628952.866124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828704384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628952.866150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628952.866167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628952.866176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628952/real 1778628952] req@ffff954358789c00 x1865013828704512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628952.866184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828704512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628952.866191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628952.866193:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628952.866222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628952.866227:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628952.866244:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628952.866252:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628952/real 1778628952] req@ffff954358788380 x1865013828704640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628952.866260:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828704640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628952.866265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628952.866268:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628952.866270:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628952.866274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628952.866284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628952.866289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628952/real 1778628952] req@ffff95435878aa00 x1865013828704768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628952.866296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828704768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628952.866302:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628952.866304:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628952.866306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628952.866319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628952.866329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628952.866343:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628952/real 1778628952] req@ffff95435878b100 x1865013828704896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628952.866351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828704896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628952.866357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628952.866359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628952.866362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828704896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628953.057785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628953.057801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628953.825778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628954.081824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628954.081917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628954.849810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628955.105777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628955.105880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628955.873750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628956.129787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628956.129871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628956.897782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628957.153779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628957.153839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628957.921802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628957.985789:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628957.985802:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628957.985808:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628957.985821:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13292 00000100:00080000:3.0:1778628957.985835:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628957.985843:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628957.985889:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828705024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628957.985921:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628957.985925:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628957.985927:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628957.985933:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13292 00000100:00080000:3.0:1778628957.985936:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628957.985940:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628957.985952:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628957.985965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828705152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628957.985978:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628957.985980:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828705024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628957.985983:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628957.985987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628957.985992:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13292 00000100:00080000:3.0:1778628957.986014:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628957.986018:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628957.986024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628957.986036:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828705280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628957.986046:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628957.986050:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628957.986052:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628957.986057:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13292 00000100:00080000:3.0:1778628957.986061:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628957.986064:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628957.986069:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628957.986071:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628957.986090:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828705408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628957.986115:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628957.986118:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628957.986121:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628957.986126:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13292 00000100:00080000:3.0:1778628957.986130:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778628957.986132:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628957/real 1778628957] req@ffff95434701c700 x1865013828705024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628957.986137:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628957.986145:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628957.986159:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828705536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628957.986163:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828705024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628957.986183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628957.986192:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628957.986199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828705024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628957.986228:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628957.986250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628957.986271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628957/real 1778628957] req@ffff95434701c000 x1865013828705152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628957.986284:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828705152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628957.986294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628957.986297:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628957.986301:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628957.986308:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628957.986326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628957.986334:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628957/real 1778628957] req@ffff95434701ea00 x1865013828705280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628957.986347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828705280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628957.986356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628957.986360:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628957.986363:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628957.986369:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628957.986387:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628957.986395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628957/real 1778628957] req@ffff95434701d180 x1865013828705408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628957.986406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828705408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628957.986420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628957.986424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628957.986427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628957.986433:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628957.986451:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628957.986460:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628957/real 1778628957] req@ffff95434701ca80 x1865013828705536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628957.986471:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828705536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628957.986480:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628957.986484:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628957.986487:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628958.177801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628958.177890:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628958.945810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628959.201785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628959.201884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628959.969780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628960.225797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628960.225875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628960.993779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778628961.249825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628961.249912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628962.017788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778628962.273852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778628962.273857:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628963.041816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628963.105815:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628963.105829:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628963.105834:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628963.105846:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13297 00000100:00080000:0.0:1778628963.105859:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628963.105866:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628963.105910:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828705664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628963.105974:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628963.105979:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628963.105981:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628963.105987:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13297 00000100:00080000:0.0:1778628963.105992:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628963.106012:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628963.106024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628963.106042:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828705792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628963.106056:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828705664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628963.106061:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628963.106071:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628963.106073:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628963.106079:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13297 00000100:00080000:0.0:1778628963.106083:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628963.106088:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628963.106094:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628963.106107:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828705920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628963.106119:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628963.106122:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628963.106124:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628963.106130:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13297 00000100:00080000:0.0:1778628963.106142:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628963.106149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628963.106152:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628963.106161:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628963.106179:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828706048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628963.106192:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628963.106197:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628963/real 1778628963] req@ffff954342f48700 x1865013828705664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628963.106199:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628963.106203:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628963.106218:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13297 00000100:00080000:0.0:1778628963.106227:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628963.106234:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628963.106240:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628963.106242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828705664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628963.106260:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828706176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628963.106293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628963.106300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628963.106306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828705664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628963.106335:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628963.106364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628963.106378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628963/real 1778628963] req@ffff954342f4b100 x1865013828705792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628963.106392:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828705792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628963.106402:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628963.106406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628963.106409:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628963.106417:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628963.106447:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628963.106460:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628963/real 1778628963] req@ffff954342f49f80 x1865013828705920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628963.106473:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828705920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628963.106481:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628963.106485:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628963.106489:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828705920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628963.106502:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628963.106525:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628963.106536:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628963/real 1778628963] req@ffff954342f49500 x1865013828706048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628963.106550:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828706048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628963.106559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628963.106562:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628963.106566:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628963.106572:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628963.106601:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628963.106612:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628963/real 1778628963] req@ffff954342f4ad80 x1865013828706176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628963.106624:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828706176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628963.106633:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628963.106636:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628963.106640:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778628963.297814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628963.297920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628964.065780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628964.321757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628964.321797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628965.089781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628965.345744:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628965.345846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628966.113743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628966.369811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628966.369905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628967.137786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628967.393771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628967.393847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628968.161753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628968.225824:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628968.225837:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628968.225842:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628968.225856:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13302 00000100:00080000:0.0:1778628968.225872:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628968.225881:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628968.225945:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828706304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628968.225971:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628968.225994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628968.225996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628968.226001:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13302 00000100:00080000:0.0:1778628968.226007:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628968.226013:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628968.226034:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628968.226036:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828706304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628968.226054:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828706432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628968.226064:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628968.226067:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628968.226069:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628968.226072:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13302 00000100:00080000:0.0:1778628968.226076:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628968.226079:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628968.226084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628968.226093:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828706560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628968.226100:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628968.226102:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628968.226104:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628968.226107:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13302 00000100:00080000:0.0:1778628968.226110:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628968.226112:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628968.226121:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628968.226131:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828706688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628968.226138:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628968.226141:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628968.226143:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628968.226146:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13302 00000100:00080000:0.0:1778628968.226149:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628968.226154:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628968.226155:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706432:192.168.204.102@tcp:38:kworker.0 00000008:00100000:0.0:1778628968.226157:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628968.226177:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828706816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628968.226198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628968.226222:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628968.226246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628968.417766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628968.417831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628969.185749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628969.441773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628969.441848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628970.209835:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628970.465771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628970.465850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628971.233821:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778628971.298071:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778628971.298125:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778628971.298157:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628971.298197:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628971.298206:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628971.298212:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778628971.298218:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778628971.298310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628968/real 1778628971] req@ffff954342f49f80 x1865013828706816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628971.298364:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828706816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628971.298377:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628971.298382:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628971.298388:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628971.298396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628968/real 1778628971] req@ffff954342f4a680 x1865013828706432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628971.298410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828706432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628971.298419:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628971.298423:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628971.298450:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628971.298459:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628968/real 1778628971] req@ffff954342f4ad80 x1865013828706560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628971.298474:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828706560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628971.298482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628971.298486:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628971.298490:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628971.298497:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628968/real 1778628971] req@ffff954342f49500 x1865013828706688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628971.298513:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828706688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628971.298524:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628971.298528:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628971.298531:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828706688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628971.298538:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628968/real 1778628971] req@ffff954342f48a80 x1865013828706304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628971.298550:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828706304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628971.298559:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628971.298563:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628971.298575:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828706304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778628971.489813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628971.489889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628972.257807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628972.513843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628972.513889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628973.281840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628973.345780:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628973.345789:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628973.345793:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628973.345813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13307 00000100:00080000:0.0:1778628973.345822:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628973.345828:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628973.345879:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828706944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628973.345901:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628973.345906:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628973.345909:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628973.345929:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13307 00000100:00080000:0.0:1778628973.345937:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628973.345941:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628973.345966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628973.345976:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828706944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628973.345981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828707072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628973.345990:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628973.345992:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628973.345993:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628973.345996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13307 00000100:00080000:0.0:1778628973.345999:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628973.346002:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628973.346007:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628973.346018:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828707200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628973.346028:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628973.346031:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628973.346033:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628973.346038:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13307 00000100:00080000:0.0:1778628973.346042:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778628973.346047:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778628973.346048:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628973.346055:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628973.346068:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828707328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628973.346086:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628973.346092:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778628973.346092:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628973/real 1778628973] req@ffff954342f4b100 x1865013828706944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778628973.346096:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628973.346106:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13307 00000100:00080000:0.0:1778628973.346112:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628973.346117:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628973.346125:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628973.346136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828706944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628973.346140:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828707456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628973.346164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628973.346172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628973.346177:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828706944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628973.346204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628973.346229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628973.346242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628973/real 1778628973] req@ffff954342f48700 x1865013828707072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628973.346255:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828707072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628973.346263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628973.346266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628973.346269:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628973.346275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628973.346308:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628973.346320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628973/real 1778628973] req@ffff954342f49c00 x1865013828707200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628973.346332:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828707200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628973.346340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628973.346343:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628973.346346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628973.346352:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628973.346374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628973.346385:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628973/real 1778628973] req@ffff954342f4b480 x1865013828707328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628973.346397:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828707328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628973.346405:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628973.346408:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628973.346411:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628973.346417:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628973.346438:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628973.346449:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628973/real 1778628973] req@ffff954342f4a300 x1865013828707456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628973.346476:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828707456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778628998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628973.346485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628973.346488:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628973.346491:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628973.537811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628973.537837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628974.305816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778628974.561816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628974.561839:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628975.329811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628975.585816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628975.585844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628976.353768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628976.609781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628976.609814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628977.377780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778628977.633829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628977.633829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778628978.401815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628978.465797:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628978.465809:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628978.465814:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628978.465823:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13312 00000100:00080000:0.0:1778628978.465834:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628978.465840:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628978.465875:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828707584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628978.465917:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628978.465920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628978.465935:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628978.465939:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13312 00000100:00080000:0.0:1778628978.465942:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628978.465945:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628978.465954:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628978.465964:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828707712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628978.465972:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628978.465974:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628978.465976:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628978.465978:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13312 00000100:00080000:0.0:1778628978.465980:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628978.465982:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628978.465985:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628978.465991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828707840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628978.465998:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628978.466002:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628978.466004:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628978.466009:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13312 00000100:00080000:0.0:1778628978.466013:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628978.466018:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628978.466024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778628978.466036:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828707584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778628978.466070:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828707968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628978.466081:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628978.466085:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628978.466087:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628978.466091:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13312 00000100:00080000:0.0:1778628978.466108:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628978.466113:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778628978.466119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778628978.466120:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778628978.466136:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828708096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628978.466164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628978/real 1778628978] req@ffff954342f49880 x1865013828707584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628978.466183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828707584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628978.466191:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628978.466197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628978.466202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828707584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628978.466211:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628978.466242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628978.466259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628978/real 1778628978] req@ffff954342f4bb80 x1865013828707712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628978.466277:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828707712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628978.466294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628978.466297:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628978.466300:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628978.466305:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628978.466338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628978.466347:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628978/real 1778628978] req@ffff954342f48e00 x1865013828707840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628978.466355:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828707840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628978.466361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628978.466364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628978.466366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628978.466386:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628978.466398:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628978.466404:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628978/real 1778628978] req@ffff954342f4aa00 x1865013828707968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628978.466412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828707968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628978.466418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628978.466420:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628978.466427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828707968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628978.466432:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628978.466446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628978.466452:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628978/real 1778628978] req@ffff954342f49180 x1865013828708096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628978.466460:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828708096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628978.466467:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628978.466468:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628978.466470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628978.657828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628978.657861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628979.425805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628979.681810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628979.681844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628980.449863:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628980.705808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628980.705866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628981.473809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628981.729800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628981.729846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628982.497816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628982.753795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628982.753867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628983.521781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778628983.585792:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628983.585810:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778628983.585815:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628983.585826:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13318 00000100:00080000:0.0:1778628983.585840:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628983.585846:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778628983.585895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828708224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628983.585924:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628983.585928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778628983.585929:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628983.585952:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13318 00000100:00080000:0.0:1778628983.585956:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628983.585960:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628983.585969:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628983.585981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828708352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628983.585990:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828708224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778628983.585997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628983.586002:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778628983.586004:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628983.586008:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13318 00000100:00080000:0.0:1778628983.586013:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628983.586017:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628983.586023:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778628983.586039:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828708480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628983.586049:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628983.586052:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778628983.586053:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628983.586057:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13318 00000100:00080000:0.0:1778628983.586060:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628983.586063:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778628983.586068:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778628983.586069:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778628983.586080:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828708608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778628983.586092:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778628983.586095:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778628983.586096:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778628983.586100:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13318 00000100:00080000:0.0:1778628983.586104:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778628983.586107:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778628983.586111:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628983.586112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628983/real 1778628983] req@ffff954342f48380 x1865013828708224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778628983.586122:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828708736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628983.586132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828708224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628983.586142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628983.586145:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628983.586150:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828708224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628983.586173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628983.586189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628983.586206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628983/real 1778628983] req@ffff954342f48000 x1865013828708352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628983.586215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828708352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628983.586221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628983.586223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628983.586226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628983.586229:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628983.586242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628983.586247:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628983/real 1778628983] req@ffff954342f49180 x1865013828708480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628983.586255:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828708480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628983.586260:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628983.586262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628983.586264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628983.586267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628983.586278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628983.586283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628983/real 1778628983] req@ffff954342f4aa00 x1865013828708608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628983.586291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828708608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628983.586301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628983.586303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628983.586305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628983.586309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628983.586322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628983.586328:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628983/real 1778628983] req@ffff954342f48e00 x1865013828708736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628983.586335:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828708736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628983.586341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628983.586344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628983.586345:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628983.777800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628983.777880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628984.545800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628984.801779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628984.801857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628985.569789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628985.825789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778628985.825843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628986.593763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628986.849772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628986.849853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628987.617764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628987.873796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628987.873880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628988.641775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628988.705799:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628988.705810:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628988.705816:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628988.705828:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13323 00000100:00080000:3.0:1778628988.705842:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628988.705849:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628988.705895:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828708864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628988.705926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628988.705930:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628988.705932:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628988.705935:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13323 00000100:00080000:3.0:1778628988.705938:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628988.705941:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628988.705950:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628988.705961:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828708992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628988.705968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628988.705971:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628988.705992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628988.705996:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13323 00000100:00080000:3.0:1778628988.706000:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628988.706003:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628988.706008:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778628988.706019:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828708864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778628988.706020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828709120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628988.706036:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628988.706039:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628988.706040:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628988.706043:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13323 00000100:00080000:3.0:1778628988.706045:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628988.706048:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628988.706051:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628988.706062:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828709248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628988.706069:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628988.706071:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628988.706073:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628988.706075:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13323 00000100:00080000:3.0:1778628988.706078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628988.706081:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628988.706085:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:1.0:1778628988.706105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628988.706105:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828709376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628988.706154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628988/real 1778628988] req@ffff95434701df80 x1865013828708864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628988.706173:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828708864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628988.706182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628988.706186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628988.706190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828708864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628988.706206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628988.706226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628988.706235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628988/real 1778628988] req@ffff95434701e680 x1865013828708992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628988.706245:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828708992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628988.706251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628988.706253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628988.706256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828708992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628988.706280:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628988.706292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628988.706298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628988/real 1778628988] req@ffff95434701f480 x1865013828709120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628988.706306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828709120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628988.706312:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628988.706315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628988.706317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628988.706321:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628988.706333:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628988.706339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628988/real 1778628988] req@ffff95434701ca80 x1865013828709248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628988.706357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828709248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628988.706368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628988.706372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628988.706376:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628988.706383:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628988.706408:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628988.706421:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628988/real 1778628988] req@ffff95434701d180 x1865013828709376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628988.706436:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828709376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628988.706447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628988.706451:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628988.706454:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628988.897792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628988.897909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628989.665834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628989.921788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628989.921823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628990.689784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628990.945784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628990.945819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628991.713789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628991.969797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778628991.969825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628992.737753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628992.993769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628992.993850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628993.761810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628993.825764:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628993.825774:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628993.825779:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628993.825790:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13328 00000100:00080000:3.0:1778628993.825802:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628993.825810:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628993.825862:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828709504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628993.825893:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628993.825896:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628993.825898:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628993.825901:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13328 00000100:00080000:3.0:1778628993.825904:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628993.825908:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628993.825918:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628993.825929:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828709632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628993.825938:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628993.825944:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828709504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628993.825960:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628993.825962:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628993.825966:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13328 00000100:00080000:3.0:1778628993.825971:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628993.825975:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628993.825981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628993.825995:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828709760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628993.826007:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628993.826011:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628993.826013:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778628993.826018:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778628993.826023:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13328 00000100:00080000:3.0:1778628993.826029:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628993.826034:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628993.826039:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778628993.826050:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828709888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628993.826059:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778628993.826063:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628993/real 1778628993] req@ffff95434701ea00 x1865013828709504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778628993.826064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628993.826067:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628993.826075:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13328 00000100:00080000:3.0:1778628993.826081:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628993.826084:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778628993.826094:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828709504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778628993.826094:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778628993.826105:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828710016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778628993.826133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628993.826141:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628993.826147:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828709504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628993.826179:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628993.826205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628993.826219:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628993/real 1778628993] req@ffff95434701c000 x1865013828709632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628993.826234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828709632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628993.826244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628993.826248:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628993.826250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628993.826256:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628993.826274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628993.826286:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628993/real 1778628993] req@ffff95434701c700 x1865013828709760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628993.826298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828709760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628993.826306:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628993.826310:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628993.826314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628993.826328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628993.826350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628993.826358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628993/real 1778628993] req@ffff95434701f800 x1865013828709888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628993.826367:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828709888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628993.826373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628993.826375:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628993.826378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828709888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628993.826382:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628993.826414:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628993.826425:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628993/real 1778628993] req@ffff95434701d880 x1865013828710016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628993.826437:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828710016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628993.826445:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628993.826449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628993.826452:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628994.017770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628994.017986:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628994.785860:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628995.041766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628995.041842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628995.809817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628996.065823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628996.065914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628996.833808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628997.089795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778628997.089888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628997.857823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778628998.113814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628998.113899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628998.881763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778628998.945766:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628998.945774:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778628998.945778:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628998.945786:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13333 00000100:00080000:3.0:1778628998.945796:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628998.945802:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778628998.945841:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828710144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628998.945864:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628998.945867:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778628998.945868:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628998.945872:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13333 00000100:00080000:3.0:1778628998.945875:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628998.945879:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628998.945888:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628998.945908:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828710272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628998.945920:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628998.945924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778628998.945926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628998.945930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13333 00000100:00100000:1.0:1778628998.945945:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828710144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778628998.945950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628998.945955:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628998.945960:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778628998.945972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828710400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628998.945981:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628998.945983:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778628998.945985:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628998.945987:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13333 00000100:00080000:3.0:1778628998.945990:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628998.945992:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628998.945996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778628998.946006:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778628998.946007:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828710528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778628998.946015:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778628998.946017:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778628998.946019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778628998.946021:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13333 00000100:00080000:3.0:1778628998.946024:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778628998.946027:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778628998.946030:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778628998.946041:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628998/real 1778628998] req@ffff95434701f100 x1865013828710144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778628998.946045:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828710656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778628998.946057:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828710144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628998.946084:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628998.946088:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628998.946092:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828710144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778628998.946097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628998.946113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628998.946120:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628998/real 1778628998] req@ffff95434701c380 x1865013828710272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628998.946128:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828710272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628998.946133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628998.946135:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628998.946137:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628998.946152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778628998.946162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628998.946168:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628998/real 1778628998] req@ffff95434701d500 x1865013828710400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628998.946187:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828710400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628998.946193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628998.946195:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628998.946197:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778628998.946201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628998.946213:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628998.946218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628998/real 1778628998] req@ffff95434701d880 x1865013828710528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628998.946225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828710528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628998.946231:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628998.946243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628998.946245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778628998.946249:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778628998.946259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778628998.946265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778628998/real 1778628998] req@ffff95434701f800 x1865013828710656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778628998.946273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828710656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778628998.946286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778628998.946303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778628998.946306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778628999.137798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778628999.137872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778628999.905811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629000.162123:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629000.162312:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629000.929780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629001.185765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629001.185970:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629001.953787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629002.210149:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629002.210336:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629002.977783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629003.233788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629003.233915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629004.001752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629004.065774:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629004.065783:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629004.065787:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629004.065795:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13338 00000100:00080000:2.0:1778629004.065804:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629004.065810:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629004.065841:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828710784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629004.065882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629004.065885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629004.065886:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629004.065889:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13338 00000100:00080000:2.0:1778629004.065892:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629004.065896:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629004.065903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629004.065913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828710912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629004.065920:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828710784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629004.065924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629004.065927:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629004.065928:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629004.065938:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13338 00000100:00080000:2.0:1778629004.065940:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629004.065943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629004.065947:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629004.065957:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828711040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629004.065964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629004.065968:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629004.065970:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629004.065975:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13338 00000800:00000100:1.0:1778629004.065978:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629004.065987:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629004.065993:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629004.065999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629004.066009:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629004/real 1778629004] req@ffff95435878bb80 x1865013828710784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629004.066016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828711168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629004.066029:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828710784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629004.066029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629004.066038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629004.066042:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629004.066046:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629004.066047:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13338 00000100:00080000:1.0:1778629004.066055:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629004.066058:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828710784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629004.066061:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629004.066066:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629004.066070:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629004.066079:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629004.066081:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828711296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629004.066102:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629004.066113:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629004/real 1778629004] req@ffff95435878b480 x1865013828710912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629004.066121:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828710912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629004.066127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629004.066129:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629004.066131:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828710912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629004.066142:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629004.066155:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629004.066162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629004/real 1778629004] req@ffff95435878b100 x1865013828711040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629004.066169:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828711040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629004.066174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629004.066176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629004.066177:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629004.066181:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629004.066206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629004.066214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629004/real 1778629004] req@ffff95435878aa00 x1865013828711168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629004.066222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828711168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629004.066227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629004.066229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629004.066231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629004.066242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629004.066253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629004.066258:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629004/real 1778629004] req@ffff954358788380 x1865013828711296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629004.066272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828711296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629004.066278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629004.066280:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629004.066282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711296:192.168.204.102@tcp:8:kworker.0 00010000:00010000:0.0:1778629004.257863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629004.257951:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629005.025731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629005.281799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629005.281887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629006.049800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629006.305827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629006.305894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629007.073732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629007.329747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629007.329783:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629008.097755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629008.353745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629008.353819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629009.121748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629009.185816:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629009.185826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629009.185831:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629009.185841:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13343 00000100:00080000:0.0:1778629009.185853:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629009.185860:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629009.185918:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828711424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629009.185947:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629009.185951:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629009.185952:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629009.185957:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13343 00000100:00080000:0.0:1778629009.185961:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629009.185965:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629009.185974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629009.185985:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828711552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629009.185995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629009.185999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629009.186001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629009.186006:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13343 00000100:00080000:0.0:1778629009.186010:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629009.186015:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629009.186020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629009.186029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828711424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629009.186034:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828711680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629009.186047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629009.186050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629009.186052:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629009.186072:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13343 00000100:00080000:0.0:1778629009.186081:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629009.186085:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629009.186103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778629009.186108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629009.186125:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828711808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629009.186138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629009.186141:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629009.186143:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778629009.186147:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629009/real 1778629009] req@ffff954342f4bb80 x1865013828711424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629009.186152:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13343 00000100:00080000:0.0:1778629009.186159:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629009.186165:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629009.186171:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629009.186171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828711424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629009.186184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629009.186186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629009.186190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828711424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629009.186197:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828711936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629009.186198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629009.186222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629009.186232:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629009/real 1778629009] req@ffff954342f49880 x1865013828711552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629009.186241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828711552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629009.186257:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629009.186259:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629009.186262:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629009.186279:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629009.186292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629009.186298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629009/real 1778629009] req@ffff954342f4a300 x1865013828711680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629009.186305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828711680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629009.186310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629009.186312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629009.186314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629009.186318:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629009.186328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629009.186333:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629009/real 1778629009] req@ffff954342f4b480 x1865013828711808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629009.186340:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828711808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629009.186344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629009.186346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629009.186352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629009.186357:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629009.186368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629009.186374:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629009/real 1778629009] req@ffff954342f49c00 x1865013828711936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629009.186381:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828711936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629009.186387:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629009.186388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629009.186390:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828711936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629009.377728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629009.377798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629010.145747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629010.401743:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629010.401806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629011.169752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629011.425760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629011.425825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629012.193731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629012.449738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629012.449786:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629013.217792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629013.473749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629013.473790:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629014.241762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629014.305747:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629014.305755:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629014.305759:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629014.305766:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13348 00000100:00080000:0.0:1778629014.305776:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629014.305781:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629014.305813:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828712064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629014.305836:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629014.305850:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629014.305851:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629014.305855:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13348 00000100:00080000:0.0:1778629014.305858:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629014.305861:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629014.305869:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629014.305880:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828712192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629014.305880:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828712064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629014.305894:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629014.305896:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629014.305897:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629014.305901:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13348 00000100:00080000:0.0:1778629014.305903:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629014.305905:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629014.305908:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629014.305918:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828712320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629014.305923:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629014.305926:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629014.305927:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629014.305930:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13348 00000100:00080000:0.0:1778629014.305943:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629014.305950:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629014.305950:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778629014.305965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629014.305979:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629014/real 1778629014] req@ffff954342f48700 x1865013828712064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629014.305981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828712448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629014.306017:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629014.306020:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828712064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629014.306024:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629014.306027:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629014.306035:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13348 00000100:00080000:1.0:1778629014.306039:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629014.306040:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629014.306049:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:1.0:1778629014.306049:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629014.306056:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828712064:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778629014.306057:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629014.306075:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828712576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629014.306082:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629014.306106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629014.306125:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629014/real 1778629014] req@ffff954342f4b100 x1865013828712192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629014.306136:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828712192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629014.306143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629014.306145:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629014.306149:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629014.306154:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629014.306166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629014.306172:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629014/real 1778629014] req@ffff954342f48a80 x1865013828712320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629014.306179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828712320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629014.306183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629014.306185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629014.306187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629014.306190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629014.306201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629014.306206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629014/real 1778629014] req@ffff954342f49500 x1865013828712448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629014.306213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828712448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629014.306223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629014.306225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629014.306227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629014.306239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629014.306251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629014.306256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629014/real 1778629014] req@ffff954342f4ad80 x1865013828712576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629014.306263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828712576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629014.306269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629014.306271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629014.306272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629014.497757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629014.497817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629015.265740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629015.521766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629015.521858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629016.289761:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629016.545734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629016.545811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629017.313752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629017.569734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629017.569781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629018.337894:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629018.593745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629018.593808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629019.361724:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629019.425787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629019.425796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629019.425800:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629019.425807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13353 00000100:00080000:2.0:1778629019.425817:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629019.425823:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629019.425855:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828712704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629019.425881:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629019.425884:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629019.425911:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629019.425915:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13353 00000100:00080000:2.0:1778629019.425920:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629019.425923:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629019.425931:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629019.425942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828712832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629019.425952:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629019.425956:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1778629019.425958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828712704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629019.425958:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629019.425965:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13353 00000100:00080000:2.0:1778629019.425970:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629019.425975:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629019.425978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629019.425984:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828712960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629019.425989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629019.425992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629019.425993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629019.425996:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13353 00000100:00080000:2.0:1778629019.425998:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629019.426000:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629019.426003:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629019.426019:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828713088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629019.426026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629019.426028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629019.426029:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629019.426031:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13353 00000100:00080000:2.0:1778629019.426034:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629019.426036:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629019.426039:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629019.426045:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828713216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629019.426062:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629019.426109:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629019/real 1778629019] req@ffff954358789c00 x1865013828712704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629019.426132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828712704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629019.426146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629019.426151:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629019.426156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828712704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629019.426200:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629019.426229:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629019.426243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629019/real 1778629019] req@ffff954358788e00 x1865013828712832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629019.426257:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828712832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629019.426267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629019.426271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629019.426275:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629019.426282:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629019.426301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629019.426311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629019/real 1778629019] req@ffff954358788700 x1865013828712960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629019.426336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828712960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629019.426344:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629019.426348:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629019.426352:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828712960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629019.426359:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629019.426379:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629019.426390:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629019/real 1778629019] req@ffff954358789880 x1865013828713088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629019.426411:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828713088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629019.426420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629019.426424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629019.426428:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629019.426435:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629019.426455:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629019.426465:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629019/real 1778629019] req@ffff954358789500 x1865013828713216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629019.426476:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828713216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629019.426484:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629019.426487:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629019.426491:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629019.617823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629019.617907:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629020.385758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629020.641784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629020.641914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629021.409836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629021.665805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629021.665912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629022.433815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629022.689790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629022.689837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629023.457747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629023.713753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629023.713843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629024.481753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629024.545779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629024.545805:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629024.545811:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629024.545820:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13359 00000100:00080000:0.0:1778629024.545830:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629024.545835:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629024.545872:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828713344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629024.545903:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629024.545907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629024.545910:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629024.545914:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13359 00000100:00080000:0.0:1778629024.545933:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629024.545936:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629024.545944:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629024.545956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828713472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629024.545960:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828713344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629024.545968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629024.545972:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629024.545973:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629024.545976:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13359 00000100:00080000:0.0:1778629024.545978:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629024.545980:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629024.545983:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629024.545992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828713600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629024.545998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629024.546000:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629024.546001:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629024.546003:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13359 00000100:00080000:0.0:1778629024.546019:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629024.546022:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629024.546025:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629024.546031:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629024.546034:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828713728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629024.546050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629024.546052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629024.546054:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629024.546056:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13359 00000100:00080000:0.0:1778629024.546060:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629024.546063:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629024.546066:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629024.546075:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629024/real 1778629024] req@ffff954342f4a680 x1865013828713344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629024.546076:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828713856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629024.546109:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828713344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629024.546135:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629024.546140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629024.546145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828713344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629024.546168:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629024.546191:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629024.546203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629024/real 1778629024] req@ffff954342f49f80 x1865013828713472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629024.546215:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828713472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629024.546224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629024.546228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629024.546231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629024.546238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629024.546254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629024.546263:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629024/real 1778629024] req@ffff954342f4ad80 x1865013828713600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629024.546276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828713600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629024.546284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629024.546288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629024.546291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629024.546303:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629024.546323:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629024.546332:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629024/real 1778629024] req@ffff954342f49500 x1865013828713728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629024.546344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828713728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629024.546353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629024.546356:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629024.546359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629024.546365:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629024.546383:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629024.546392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629024/real 1778629024] req@ffff954342f48a80 x1865013828713856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629024.546404:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828713856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629024.546412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629024.546415:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629024.546418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828713856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629024.737774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629024.737846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629025.505794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629025.761807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629025.761887:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629026.529767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629026.785780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629026.785886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629027.553750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629027.809755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629027.809822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629028.577749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629028.833769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629028.833835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629029.601771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629029.665824:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629029.665832:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629029.665836:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629029.665844:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13364 00000100:00080000:2.0:1778629029.665854:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629029.665859:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629029.665894:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828713984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629029.665915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629029.665918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629029.665919:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629029.665922:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13364 00000100:00080000:2.0:1778629029.665924:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629029.665927:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629029.665934:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629029.665945:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828714112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629029.665958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828713984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629029.665960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629029.665966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629029.665967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629029.665970:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13364 00000100:00080000:2.0:1778629029.665972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629029.665987:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629029.665992:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629029.666001:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828714240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629029.666008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629029.666010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629029.666011:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629029.666014:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13364 00000800:00000100:1.0:1778629029.666018:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629029.666019:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629029.666024:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629029.666028:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629029.666037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828714368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629029.666044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629029.666047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629029.666048:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629029.666051:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13364 00000100:00080000:2.0:1778629029.666053:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629029.666055:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629029/real 1778629029] req@ffff95435878b800 x1865013828713984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629029.666057:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629029.666060:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629029.666068:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828714496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629029.666085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828713984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629029.666095:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629029.666099:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629029.666102:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828713984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629029.666121:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629029.666138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629029.666145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629029/real 1778629029] req@ffff95435878a300 x1865013828714112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629029.666153:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828714112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629029.666159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629029.666161:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629029.666163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629029.666166:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629029.666176:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629029.666181:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629029/real 1778629029] req@ffff95435878ad80 x1865013828714240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629029.666188:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828714240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629029.666205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629029.666209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629029.666212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629029.666218:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629029.666240:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629029.666251:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629029/real 1778629029] req@ffff954358789180 x1865013828714368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629029.666263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828714368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629029.666269:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629029.666271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629029.666273:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629029.666277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629029.666291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629029.666297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629029/real 1778629029] req@ffff954358788a80 x1865013828714496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629029.666304:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828714496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629029.666309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629029.666311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629029.666318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629029.857764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629029.857834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629030.625741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629030.881745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629030.881808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629031.649754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629031.905775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629031.905849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629032.673762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629032.929763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629032.929833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629033.697779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629033.953783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629033.953861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629034.721806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629034.785840:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629034.785848:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629034.785852:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629034.785860:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13369 00000100:00080000:1.0:1778629034.785870:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629034.785876:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629034.785908:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828714624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629034.785936:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629034.785939:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629034.785940:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629034.785943:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13369 00000100:00080000:1.0:1778629034.785945:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629034.785947:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629034.785955:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629034.785963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828714752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629034.785970:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629034.785973:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629034.785974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629034.785976:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13369 00000100:00080000:1.0:1778629034.785979:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778629034.785985:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828714624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629034.785996:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629034.786001:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629034.786011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828714880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629034.786018:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629034.786020:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629034.786022:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629034.786024:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13369 00000100:00080000:1.0:1778629034.786035:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629034.786039:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629034.786042:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629034.786049:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828715008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629034.786056:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629034.786058:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629034.786060:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629034.786063:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13369 00000100:00080000:1.0:1778629034.786065:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629034.786068:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629034.786072:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714752:192.168.204.102@tcp:38:kworker.0 00000008:00100000:1.0:1778629034.786074:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629034.786086:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828715136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778629034.786109:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629034.786138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629034.786156:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629034.977794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629034.977847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629035.745815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629036.001758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629036.001816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629036.769768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629037.025771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629037.025825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629037.793858:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778629037.857980:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778629037.858031:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778629037.858049:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629037.858092:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629037.858103:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629037.858115:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629037.858127:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:0.0:1778629037.858152:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629034/real 1778629037] req@ffff954350195c00 x1865013828714880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629037.858186:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828714880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629037.858200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629037.858208:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629037.858235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629037.858244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629034/real 1778629037] req@ffff954350195500 x1865013828715008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629037.858266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828715008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629037.858275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629037.858279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629037.858283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629037.858291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629034/real 1778629037] req@ffff954350195f80 x1865013828715136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629037.858302:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828715136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629037.858313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629037.858316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629037.858319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629037.858325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629034/real 1778629037] req@ffff954350194380 x1865013828714624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629037.858338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828714624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629037.858346:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629037.858348:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629037.858359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828714624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629037.858367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629034/real 1778629037] req@ffff954350195180 x1865013828714752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629037.858380:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828714752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629037.858390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629037.858394:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629037.858397:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828714752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629038.049806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629038.049874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629038.817771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629039.073765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629039.073825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629039.841803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629039.905781:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629039.905811:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629039.905816:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629039.905827:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13374 00000100:00080000:1.0:1778629039.905840:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629039.905847:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629039.905887:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828715264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629039.905916:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629039.905919:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629039.905920:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629039.905924:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13374 00000100:00080000:1.0:1778629039.905926:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629039.905929:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629039.905941:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629039.905957:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828715392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778629039.905968:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828715264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629039.905975:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629039.905981:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629039.905983:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629039.905988:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13374 00000100:00080000:1.0:1778629039.905995:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629039.906003:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629039.906011:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629039.906032:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828715520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629039.906051:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778629039.906060:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629039.906064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629039.906068:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629039.906078:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13374 00000100:00080000:1.0:1778629039.906087:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629039.906093:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629039.906099:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629039.906111:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629039/real 1778629039] req@ffff954350195880 x1865013828715264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629039.906120:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828715648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629039.906156:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629039.906162:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629039.906164:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629039.906175:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13374 00000100:00080000:1.0:1778629039.906180:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778629039.906183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828715264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629039.906191:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629039.906203:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778629039.906221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629039.906232:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629039.906236:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828715776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778629039.906240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828715264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629039.906315:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629039.906346:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629039.906363:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629039/real 1778629039] req@ffff954350195180 x1865013828715392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629039.906418:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828715392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629039.906428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629039.906432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629039.906436:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629039.906443:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629039.906468:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629039.906478:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629039/real 1778629039] req@ffff954350194380 x1865013828715520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629039.906507:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828715520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629039.906514:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629039.906518:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629039.906522:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629039.906529:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629039.906552:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629039.906564:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629039/real 1778629039] req@ffff954350195f80 x1865013828715648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629039.906575:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828715648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629039.906593:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629039.906597:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629039.906600:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629039.906607:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629039.906628:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629039.906638:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629039/real 1778629039] req@ffff954350195500 x1865013828715776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629039.906648:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828715776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629039.906656:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629039.906659:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629039.906662:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828715776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629040.097779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629040.097884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629040.865773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629041.121790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629041.121851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629041.889787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629042.145777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629042.145838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629042.913774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629043.169767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629043.169839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629043.937772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629044.193816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629044.193843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629044.961783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629045.025832:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629045.025845:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629045.025851:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629045.025864:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13379 00000100:00080000:1.0:1778629045.025901:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629045.025910:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629045.025957:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828715904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629045.025993:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629045.025997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629045.025999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629045.026004:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13379 00000100:00080000:1.0:1778629045.026008:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629045.026012:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629045.026024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629045.026041:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828716032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629045.026054:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629045.026058:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629045.026060:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629045.026067:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13379 00000100:00080000:1.0:1778629045.026076:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778629045.026078:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828715904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629045.026087:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629045.026095:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629045.026115:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828716160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629045.026128:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629045.026143:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629045.026145:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1778629045.026150:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629045.026154:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13379 00000100:00080000:1.0:1778629045.026162:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629045.026168:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629045.026175:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629045.026193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629045/real 1778629045] req@ffff954350195c00 x1865013828715904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629045.026197:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828716288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629045.026240:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778629045.026241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828715904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629045.026246:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629045.026249:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629045.026262:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13379 00000100:00080000:1.0:1778629045.026272:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629045.026278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629045.026279:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629045.026293:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:0.0:1778629045.026295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629045.026304:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828715904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629045.026313:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828716416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778629045.026336:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629045.026363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629045.026378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629045/real 1778629045] req@ffff954350194a80 x1865013828716032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629045.026405:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828716032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629045.026412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629045.026414:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629045.026417:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629045.026445:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629045.026460:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629045.026468:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629045/real 1778629045] req@ffff954350194e00 x1865013828716160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629045.026476:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828716160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629045.026481:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629045.026484:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629045.026486:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629045.026490:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629045.026501:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629045.026506:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629045/real 1778629045] req@ffff954350196a00 x1865013828716288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629045.026521:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828716288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629045.026527:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629045.026529:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629045.026531:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629045.026535:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629045.026547:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629045.026553:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629045/real 1778629045] req@ffff954350197b80 x1865013828716416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629045.026561:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828716416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629045.026566:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629045.026568:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629045.026570:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629045.217776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629045.217876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629045.985809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629046.241811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629046.241882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629047.009813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629047.265803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629047.265807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629048.033779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629048.289803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629048.289856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629049.057773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629049.313766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629049.313825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629050.081766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629050.145791:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629050.145801:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629050.145806:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629050.145815:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13384 00000100:00080000:0.0:1778629050.145827:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629050.145834:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629050.145871:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828716544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629050.145913:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629050.145917:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629050.145919:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629050.145924:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13384 00000100:00080000:0.0:1778629050.145929:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629050.145934:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629050.145944:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629050.145952:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828716544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629050.145960:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828716672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629050.145973:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629050.145977:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629050.145978:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629050.145983:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13384 00000100:00080000:0.0:1778629050.145987:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629050.146001:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629050.146008:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778629050.146019:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629050.146022:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828716800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629050.146038:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629050.146042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629050.146044:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629050.146049:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13384 00000100:00080000:0.0:1778629050.146060:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629050.146067:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629050.146067:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629050/real 1778629050] req@ffff954342f4b100 x1865013828716544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778629050.146075:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629050.146092:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828716928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629050.146103:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629050.146107:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778629050.146107:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828716544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629050.146112:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629050.146121:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13384 00000100:00080000:0.0:1778629050.146125:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629050.146130:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629050.146138:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778629050.146144:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778629050.146152:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828717056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629050.146162:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629050.146171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828716544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629050.146197:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629050.146230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629050.146245:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629050/real 1778629050] req@ffff954342f48700 x1865013828716672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629050.146256:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828716672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629050.146266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629050.146269:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629050.146272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629050.146278:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629050.146312:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629050.146322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629050/real 1778629050] req@ffff954342f49c00 x1865013828716800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629050.146335:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828716800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629050.146342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629050.146345:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629050.146348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629050.146363:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629050.146385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629050.146395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629050/real 1778629050] req@ffff954342f4b480 x1865013828716928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629050.146405:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828716928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629050.146412:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629050.146415:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629050.146418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828716928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629050.146423:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629050.146441:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629050.146450:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629050/real 1778629050] req@ffff954342f4a300 x1865013828717056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629050.146460:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828717056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629050.146469:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629050.146472:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629050.146475:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629050.337738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629050.337824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629051.105746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629051.361775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629051.361846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629052.129778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629052.385772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629052.385831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629053.153811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629053.409748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629053.409820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629054.177811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629054.433792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629054.433864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629055.201763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629055.265772:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629055.265779:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629055.265782:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629055.265805:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13389 00000100:00080000:1.0:1778629055.265813:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629055.265819:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629055.265845:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828717184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629055.265866:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629055.265868:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629055.265870:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629055.265872:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13389 00000100:00080000:1.0:1778629055.265874:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629055.265877:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629055.265884:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629055.265892:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828717312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629055.265899:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1778629055.265903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828717184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629055.265903:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629055.265904:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629055.265908:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13389 00000100:00080000:1.0:1778629055.265912:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629055.265915:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629055.265917:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629055.265923:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828717440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629055.265936:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629055.265938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629055.265939:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629055.265942:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13389 00000100:00080000:1.0:1778629055.265945:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629055.265947:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629055.265951:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629055.265960:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828717568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629055.265968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1778629055.265972:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629055.265975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629055.265977:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629055.265982:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13389 00000100:00080000:1.0:1778629055.265987:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629055.265991:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629055.265996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629055.266002:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629055/real 1778629055] req@ffff954350194700 x1865013828717184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629055.266008:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828717696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778629055.266019:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828717184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629055.266027:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629055.266031:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629055.266034:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828717184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629055.266056:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629055.266071:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629055.266078:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629055/real 1778629055] req@ffff954350197800 x1865013828717312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629055.266085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828717312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629055.266090:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629055.266092:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629055.266094:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629055.266097:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629055.266105:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629055.266110:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629055/real 1778629055] req@ffff954350197480 x1865013828717440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629055.266117:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828717440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629055.266121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629055.266122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629055.266124:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629055.266127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629055.266136:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629055.266140:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629055/real 1778629055] req@ffff954350196d80 x1865013828717568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629055.266151:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828717568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629055.266156:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629055.266157:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629055.266159:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629055.266162:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629055.266172:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629055.266176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629055/real 1778629055] req@ffff954350194000 x1865013828717696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629055.266196:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828717696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629055.266202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629055.266204:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629055.266205:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629055.457770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629055.457829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629056.225744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629056.481735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629056.481785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629057.249791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629057.505732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629057.505801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629058.273836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629058.529787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629058.529821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629059.297816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629059.553809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629059.553884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629060.321779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629060.385763:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629060.385778:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629060.385782:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629060.385792:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13394 00000100:00080000:0.0:1778629060.385802:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629060.385809:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629060.385852:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828717824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629060.385891:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629060.385894:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629060.385896:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629060.385899:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13394 00000100:00080000:0.0:1778629060.385903:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629060.385906:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629060.385915:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629060.385920:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828717824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629060.385927:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828717952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629060.385935:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629060.385937:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629060.385938:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629060.385940:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13394 00000100:00080000:0.0:1778629060.385942:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629060.385944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629060.385947:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629060.385954:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828718080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629060.385960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629060.385962:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629060.385963:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629060.385965:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13394 00000100:00080000:0.0:1778629060.385973:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629060.385976:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629060.385980:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629060.385984:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629060.385996:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828718208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629060.386003:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629060.386005:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629060.386006:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629060.386009:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13394 00000100:00080000:0.0:1778629060.386011:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629060.386013:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629060.386015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629060.386023:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828718336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629060.386034:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629060/real 1778629060] req@ffff954342f49880 x1865013828717824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629060.386063:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828717824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629060.386071:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629060.386074:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629060.386087:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828717824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629060.386110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629060.386127:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629060.386144:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629060/real 1778629060] req@ffff954342f4bb80 x1865013828717952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629060.386155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828717952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629060.386163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629060.386167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629060.386170:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828717952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629060.386176:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629060.386195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629060.386202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629060/real 1778629060] req@ffff954342f48e00 x1865013828718080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629060.386214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828718080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629060.386232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629060.386236:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629060.386239:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629060.386245:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629060.386270:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629060.386276:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629060/real 1778629060] req@ffff954342f4aa00 x1865013828718208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629060.386283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828718208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629060.386289:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629060.386291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629060.386293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629060.386296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629060.386306:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629060.386310:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629060/real 1778629060] req@ffff954342f49180 x1865013828718336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629060.386317:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828718336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629060.386322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629060.386323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629060.386329:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629060.577786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629060.577855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629061.345752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629061.601761:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629061.601817:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629062.369748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629062.625773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629062.625856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629063.393818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629063.649812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629063.649898:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629064.417729:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629064.673788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629064.673858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629065.441767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629065.505847:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629065.505859:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629065.505864:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629065.505876:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13400 00000100:00080000:0.0:1778629065.505890:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629065.505899:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629065.505949:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828718464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629065.505990:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629065.506015:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629065.506017:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629065.506021:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13400 00000100:00080000:0.0:1778629065.506024:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629065.506028:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629065.506038:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629065.506042:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828718464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629065.506056:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828718592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629065.506070:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629065.506074:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629065.506077:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629065.506082:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13400 00000100:00080000:0.0:1778629065.506105:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629065.506109:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629065.506114:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778629065.506115:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629065.506131:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828718720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629065.506139:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629065.506142:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629065.506144:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629065.506148:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13400 00000100:00080000:0.0:1778629065.506152:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629065.506156:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629065.506159:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629065/real 1778629065] req@ffff954342f48000 x1865013828718464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778629065.506169:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629065.506182:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828718464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629065.506183:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828718848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629065.506207:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629065.506207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629065.506213:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629065.506215:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629065.506216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778629065.506223:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13400 00000100:00100000:1.0:1778629065.506225:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828718464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629065.506227:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629065.506234:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629065.506237:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629065.506244:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828718976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629065.506254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629065.506274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629065.506285:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629065/real 1778629065] req@ffff954342f48380 x1865013828718592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629065.506296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828718592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629065.506303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629065.506306:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629065.506309:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629065.506315:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629065.506330:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629065.506347:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629065/real 1778629065] req@ffff954342f49180 x1865013828718720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629065.506356:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828718720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629065.506362:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629065.506364:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629065.506366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629065.506370:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629065.506382:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629065.506387:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629065/real 1778629065] req@ffff954342f4aa00 x1865013828718848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629065.506394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828718848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629065.506399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629065.506401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629065.506403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629065.506414:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629065.506424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629065.506430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629065/real 1778629065] req@ffff954342f48e00 x1865013828718976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629065.506437:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828718976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629065.506447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629065.506450:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629065.506452:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828718976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629065.697807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629065.697873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629066.465759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629066.721890:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629066.722121:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629067.489825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629067.745885:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629067.746278:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629068.513803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629068.769854:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629068.770024:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629069.537755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629069.794527:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629069.794804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629070.561736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629070.625864:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629070.625872:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629070.625911:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629070.625920:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13405 00000100:00080000:3.0:1778629070.625930:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629070.625936:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629070.625965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828719104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629070.625992:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629070.625995:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629070.625996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629070.625999:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13405 00000100:00080000:3.0:1778629070.626002:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629070.626004:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629070.626011:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629070.626020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828719232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629070.626027:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629070.626030:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629070.626031:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629070.626033:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13405 00000100:00080000:3.0:1778629070.626035:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629070.626037:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629070.626044:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828719104:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778629070.626045:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629070.626057:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828719360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629070.626075:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629070.626077:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629070.626079:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629070.626082:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13405 00000100:00080000:3.0:1778629070.626091:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629070.626094:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629070.626098:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629070.626106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778629070.626109:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828719488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629070.626116:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629070.626118:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629070.626120:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629070.626122:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13405 00000100:00080000:3.0:1778629070.626131:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629070.626133:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629070.626137:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629070.626141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629070/real 1778629070] req@ffff95434701c700 x1865013828719104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629070.626146:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828719616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629070.626171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828719104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629070.626180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629070.626184:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629070.626188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828719104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629070.626193:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629070.626206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629070.626213:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629070/real 1778629070] req@ffff95434701c000 x1865013828719232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629070.626220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828719232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629070.626226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629070.626228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629070.626230:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629070.626254:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629070.626267:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629070.626273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629070/real 1778629070] req@ffff95434701ea00 x1865013828719360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629070.626281:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828719360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629070.626286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629070.626288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629070.626290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629070.626293:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629070.626303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629070.626308:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629070/real 1778629070] req@ffff95434701d180 x1865013828719488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629070.626314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828719488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629070.626320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629070.626321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629070.626323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629070.626326:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629070.626335:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629070.626340:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629070/real 1778629070] req@ffff95434701ca80 x1865013828719616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629070.626352:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828719616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629070.626358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629070.626359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629070.626361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629070.818129:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629070.818427:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629071.585740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629071.841848:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629071.842428:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629072.609732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629072.865777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629072.865866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629073.633794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629073.889808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629073.889892:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629074.657808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629074.913821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629074.913864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629075.681788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629075.745864:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629075.745877:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629075.745883:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629075.745896:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13410 00000100:00080000:0.0:1778629075.745912:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629075.745920:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629075.745984:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828719744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629075.746023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629075.746028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629075.746030:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629075.746035:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13410 00000100:00080000:0.0:1778629075.746060:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629075.746066:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629075.746078:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629075.746080:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828719744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629075.746096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828719872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629075.746109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629075.746113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629075.746115:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629075.746120:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13410 00000100:00080000:0.0:1778629075.746142:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629075.746146:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629075.746152:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778629075.746153:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629075.746168:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828720000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629075.746178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629075.746182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629075.746184:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629075.746189:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13410 00000100:00100000:1.0:1778629075.746192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629075/real 1778629075] req@ffff954342f4bb80 x1865013828719744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629075.746195:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629075.746204:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629075.746210:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629075.746219:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828719744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629075.746224:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828720128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629075.746233:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629075.746238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629075.746242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828719744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629075.746257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629075.746260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629075.746263:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629075.746269:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13410 00000100:00100000:1.0:1778629075.746272:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719872:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778629075.746275:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629075.746282:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629075.746287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:1.0:1778629075.746293:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629075.746303:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828720256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629075.746311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629075/real 1778629075] req@ffff954342f49880 x1865013828719872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629075.746330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828719872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629075.746356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629075.746361:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629075.746365:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828719872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629075.746372:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629075.746395:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629075.746406:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629075/real 1778629075] req@ffff954342f4a300 x1865013828720000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629075.746414:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828720000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629075.746420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629075.746422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629075.746424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629075.746428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629075.746441:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629075.746447:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629075/real 1778629075] req@ffff954342f4b480 x1865013828720128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629075.746455:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828720128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629075.746459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629075.746461:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629075.746468:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629075.746483:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629075.746504:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629075.746514:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629075/real 1778629075] req@ffff954342f49c00 x1865013828720256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629075.746527:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828720256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629075.746536:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629075.746539:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629075.746543:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629075.937812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629075.937880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629076.705794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629076.961839:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629076.961912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629077.729801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629077.985805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629077.985912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629078.753778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629079.009826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629079.009936:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629079.777809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629080.033796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629080.033886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629080.801787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629080.865756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629080.865764:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629080.865768:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629080.865777:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13415 00000100:00080000:2.0:1778629080.865787:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629080.865793:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629080.865824:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828720384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629080.865845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629080.865848:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629080.865849:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629080.865852:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13415 00000100:00080000:2.0:1778629080.865855:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629080.865857:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629080.865865:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629080.865873:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828720512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629080.865879:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828720384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629080.865900:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629080.865903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629080.865904:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629080.865908:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13415 00000100:00080000:2.0:1778629080.865911:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629080.865914:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629080.865917:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629080.865927:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828720640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629080.865933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629080.865935:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629080.865937:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629080.865939:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13415 00000100:00080000:2.0:1778629080.865942:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629080.865944:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629080.865947:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629080.865953:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629080.865956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828720768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629080.865963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629080.865965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629080.865967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629080.865970:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13415 00000100:00080000:2.0:1778629080.865977:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629080.865981:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629080.865985:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629080.865986:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629080/real 1778629080] req@ffff95435878a680 x1865013828720384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629080.866000:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828720896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629080.866010:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828720384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629080.866020:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629080.866024:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629080.866028:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828720384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629080.866047:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629080.866063:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629080.866079:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629080/real 1778629080] req@ffff954358789f80 x1865013828720512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629080.866087:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828720512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629080.866094:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629080.866096:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629080.866098:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629080.866102:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629080.866114:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629080.866119:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629080/real 1778629080] req@ffff954358788a80 x1865013828720640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629080.866127:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828720640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629080.866131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629080.866133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629080.866135:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629080.866139:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629080.866149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629080.866154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629080/real 1778629080] req@ffff954358789180 x1865013828720768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629080.866161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828720768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629080.866170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629080.866172:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629080.866175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629080.866178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629080.866190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629080.866195:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629080/real 1778629080] req@ffff95435878ad80 x1865013828720896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629080.866202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828720896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629080.866208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629080.866210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629080.866211:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828720896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629081.057777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629081.057849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629081.825773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629082.081819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629082.081927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629082.849843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629083.105831:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629083.105894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629083.873849:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629084.129823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629084.129893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629084.897846:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629085.153835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629085.153902:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629085.921863:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629085.985843:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629085.985852:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629085.985856:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629085.985865:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13420 00000100:00080000:3.0:1778629085.985875:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629085.985881:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629085.985913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828721024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629085.985939:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629085.985942:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629085.985943:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629085.985946:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13420 00000100:00080000:3.0:1778629085.985949:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629085.985951:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629085.985959:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629085.985969:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828721152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629085.985977:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629085.985992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1778629085.985994:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828721024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629085.985995:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629085.986001:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13420 00000100:00080000:3.0:1778629085.986006:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629085.986018:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629085.986023:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629085.986034:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828721280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629085.986042:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629085.986044:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629085.986045:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629085.986048:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13420 00000100:00080000:3.0:1778629085.986052:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629085.986054:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629085.986057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629085.986065:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778629085.986069:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828721408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629085.986076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629085.986079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629085.986081:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629085.986083:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13420 00000100:00080000:3.0:1778629085.986086:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629085.986089:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629085.986103:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629085.986108:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629085/real 1778629085] req@ffff95434701f480 x1865013828721024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629085.986115:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828721536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629085.986147:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828721024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629085.986160:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629085.986165:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629085.986178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828721024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629085.986186:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629085.986222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629085.986235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629085/real 1778629085] req@ffff95434701e680 x1865013828721152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629085.986247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828721152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629085.986256:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629085.986259:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629085.986262:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629085.986286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629085.986305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629085.986316:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629085/real 1778629085] req@ffff95434701df80 x1865013828721280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629085.986327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828721280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629085.986335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629085.986338:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629085.986341:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629085.986347:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629085.986364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629085.986382:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629085/real 1778629085] req@ffff95434701ca80 x1865013828721408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629085.986393:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828721408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629085.986401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629085.986404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629085.986407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629085.986413:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629085.986434:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629085.986455:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629085/real 1778629085] req@ffff95434701d180 x1865013828721536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629085.986468:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828721536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629085.986477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629085.986480:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629085.986484:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629086.177815:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629086.177883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629086.945784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629087.201799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629087.201860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629087.969765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629088.225749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629088.225803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629088.993738:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629089.249759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629089.249804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629090.017780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629090.273767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629090.273847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629091.041771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629091.105835:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629091.105847:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629091.105852:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629091.105864:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13425 00000100:00080000:3.0:1778629091.105877:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629091.105890:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629091.105944:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828721664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629091.105985:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629091.105992:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629091.105994:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629091.105999:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13425 00000100:00080000:3.0:1778629091.106003:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629091.106007:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629091.106018:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629091.106030:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828721792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629091.106043:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828721664:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629091.106065:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629091.106069:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629091.106071:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629091.106075:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13425 00000100:00080000:3.0:1778629091.106079:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629091.106083:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629091.106088:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629091.106099:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828721920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629091.106108:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629091.106111:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629091.106113:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629091.106118:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13425 00000800:00000100:1.0:1778629091.106119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629091.106124:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629091.106129:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629091.106134:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629091.106146:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828722048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629091.106155:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629091.106158:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778629091.106165:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629091/real 1778629091] req@ffff95434701ea00 x1865013828721664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629091.106166:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629091.106175:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13425 00000100:00080000:3.0:1778629091.106185:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629091.106192:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828721664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629091.106192:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629091.106197:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629091.106208:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828722176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629091.106213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629091.106221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629091.106227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828721664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629091.106255:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629091.106278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629091.106289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629091/real 1778629091] req@ffff95434701c000 x1865013828721792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629091.106301:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828721792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629091.106310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629091.106313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629091.106316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629091.106322:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629091.106340:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629091.106351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629091/real 1778629091] req@ffff95434701c700 x1865013828721920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629091.106361:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828721920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629091.106369:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629091.106372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629091.106375:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828721920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629091.106386:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629091.106404:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629091.106413:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629091/real 1778629091] req@ffff95434701f800 x1865013828722048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629091.106424:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828722048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629091.106432:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629091.106435:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629091.106438:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629091.106443:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629091.106467:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629091.106489:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629091/real 1778629091] req@ffff95434701d880 x1865013828722176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629091.106500:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828722176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629091.106509:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629091.106512:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629091.106515:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629091.297796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629091.297878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629092.065789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629092.321756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629092.321830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629093.089781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629093.345728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629093.345798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629094.113831:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629094.369796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629094.369855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629095.137824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629095.393772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629095.393833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629096.161784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629096.225812:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629096.225822:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629096.225827:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629096.225837:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13430 00000100:00080000:0.0:1778629096.225849:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629096.225855:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629096.225896:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828722304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629096.225927:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629096.225930:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629096.225931:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629096.225934:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13430 00000100:00080000:0.0:1778629096.225936:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629096.225939:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629096.225964:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629096.225981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828722432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629096.225991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629096.225994:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629096.225996:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629096.225999:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13430 00000100:00080000:0.0:1778629096.226002:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629096.226006:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629096.226010:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629096.226020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828722560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629096.226029:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629096.226032:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629096.226034:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629096.226039:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13430 00000100:00080000:0.0:1778629096.226044:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629096.226048:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629096.226053:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629096.226063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828722688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629096.226075:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629096.226078:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828722304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629096.226082:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629096.226091:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629096.226098:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13430 00000100:00080000:0.0:1778629096.226103:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629096.226108:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629096.226115:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629096.226130:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828722816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629096.226145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629096.226177:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629096/real 1778629096] req@ffff954342f48700 x1865013828722304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629096.226201:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828722304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629096.226209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629096.226213:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629096.226215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828722304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629096.226221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629096.226236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629096.226242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629096/real 1778629096] req@ffff954342f4b100 x1865013828722432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629096.226249:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828722432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629096.226254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629096.226262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629096.226265:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629096.226268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629096.226278:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629096.226283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629096/real 1778629096] req@ffff954342f48a80 x1865013828722560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629096.226290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828722560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629096.226301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629096.226303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629096.226305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629096.226324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629096.226337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629096.226342:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629096/real 1778629096] req@ffff954342f49500 x1865013828722688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629096.226349:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828722688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629096.226353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629096.226355:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629096.226357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629096.226360:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629096.226368:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629096.226384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629096/real 1778629096] req@ffff954342f4ad80 x1865013828722816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629096.226391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828722816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629096.226396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629096.226398:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629096.226403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828722816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629096.417725:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629096.417829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629097.185773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629097.441740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629097.441843:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629098.209829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629098.465760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629098.465821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629099.233794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629099.489800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629099.489828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629100.257786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629100.513796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629100.513820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629101.281819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629101.345788:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629101.345797:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629101.345801:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629101.345808:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13435 00000100:00080000:3.0:1778629101.345818:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629101.345825:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629101.345859:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828722944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629101.345887:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629101.345890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629101.345892:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629101.345896:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13435 00000100:00080000:3.0:1778629101.345899:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629101.345902:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629101.345922:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629101.345935:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828723072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629101.345944:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629101.345948:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629101.345950:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629101.345954:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13435 00000100:00080000:3.0:1778629101.345958:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629101.345962:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629101.345967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629101.345979:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828723200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629101.345981:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828722944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629101.346003:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629101.346005:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629101.346006:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629101.346010:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13435 00000100:00080000:3.0:1778629101.346013:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629101.346018:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629101.346022:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629101.346037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828723328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629101.346047:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629101.346050:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629101.346060:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629101.346066:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13435 00000100:00080000:3.0:1778629101.346070:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629101.346073:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629101.346077:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629101.346087:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828723456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629101.346110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629101.346142:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629101.346156:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629101.346169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629101.537782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629101.537805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629102.305828:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629102.561790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629102.561821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629103.329819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629103.585791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629103.585819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629104.353782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778629104.417969:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778629104.418020:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778629104.418040:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629104.418078:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629104.418085:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629104.418092:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629104.418098:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778629104.418135:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629101/real 1778629104] req@ffff95434701f100 x1865013828723200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629104.418166:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828723200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629104.418182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629104.418190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629104.418196:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629104.418223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629101/real 1778629104] req@ffff95434701d880 x1865013828723328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629104.418242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828723328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629104.418251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629104.418256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629104.418260:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629104.418291:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629101/real 1778629104] req@ffff95434701f800 x1865013828723456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629104.418305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828723456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629104.418313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629104.418317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629104.418320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629104.418327:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629101/real 1778629104] req@ffff95434701d500 x1865013828722944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629104.418339:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828722944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629104.418347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629104.418351:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629104.418354:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828722944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629104.418362:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629101/real 1778629104] req@ffff95434701c380 x1865013828723072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629104.418375:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828723072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629104.418384:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629104.418388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629104.418391:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629104.609800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629104.609832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629105.377789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629105.633804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629105.633879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629106.401775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629106.465808:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629106.465818:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629106.465823:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629106.465832:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13440 00000100:00080000:0.0:1778629106.465844:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629106.465852:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629106.465897:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828723584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629106.465939:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629106.465943:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629106.465946:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629106.465951:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13440 00000100:00080000:0.0:1778629106.465974:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629106.465979:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629106.465994:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629106.466003:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828723584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629106.466009:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828723712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629106.466020:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629106.466023:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629106.466025:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629106.466028:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13440 00000100:00080000:0.0:1778629106.466030:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629106.466034:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629106.466038:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629106.466052:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828723840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629106.466063:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629106.466067:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629106.466069:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629106.466073:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13440 00000100:00080000:0.0:1778629106.466088:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629106.466094:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629106.466100:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629106.466108:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629106.466115:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828723968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629106.466145:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629106.466149:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629106.466151:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629106.466157:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13440 00000100:00100000:1.0:1778629106.466162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629106/real 1778629106] req@ffff954342f49f80 x1865013828723584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629106.466165:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629106.466186:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629106.466196:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629106.466211:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828723584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629106.466214:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828724096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629106.466249:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629106.466258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629106.466263:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828723584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629106.466295:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629106.466322:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629106.466335:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629106/real 1778629106] req@ffff954342f4a680 x1865013828723712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629106.466347:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828723712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629106.466356:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629106.466359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629106.466362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629106.466368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629106.466389:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629106.466400:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629106/real 1778629106] req@ffff954342f4ad80 x1865013828723840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629106.466430:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828723840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629106.466439:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629106.466442:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629106.466445:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629106.466452:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629106.466477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629106.466489:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629106/real 1778629106] req@ffff954342f49500 x1865013828723968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629106.466502:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828723968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629106.466511:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629106.466515:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629106.466518:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828723968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629106.466524:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629106.466551:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629106.466564:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629106/real 1778629106] req@ffff954342f48a80 x1865013828724096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629106.466575:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828724096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629106.466596:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629106.466600:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629106.466604:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629106.657795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629106.657884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629107.425781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629107.681790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629107.681858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629108.449802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629108.705817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629108.705900:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629109.473782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629109.729808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629109.729927:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629110.497753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629110.753774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629110.753886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629111.521797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629111.585858:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629111.585870:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629111.585875:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629111.585886:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13446 00000100:00080000:1.0:1778629111.585899:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629111.585906:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629111.585951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828724224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629111.585979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629111.585982:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629111.585984:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629111.585987:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13446 00000100:00080000:1.0:1778629111.585990:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629111.585993:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629111.586003:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629111.586016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828724224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629111.586019:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828724352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629111.586048:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629111.586051:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629111.586053:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629111.586056:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13446 00000100:00080000:1.0:1778629111.586096:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629111.586100:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629111.586105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778629111.586107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629111.586120:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828724480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629111.586128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629111.586131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629111.586133:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629111.586136:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13446 00000100:00080000:1.0:1778629111.586140:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629111.586143:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629111.586147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629111.586151:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629111/real 1778629111] req@ffff954350197100 x1865013828724224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629111.586158:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828724608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629111.586168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629111.586171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1778629111.586174:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828724224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629111.586174:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629111.586183:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13446 00000100:00080000:0.0:1778629111.586189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629111.586192:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629111.586195:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629111.586199:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629111.586202:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828724224:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778629111.586206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629111.586218:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828724736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778629111.586229:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629111.586248:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629111.586256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629111/real 1778629111] req@ffff954350194000 x1865013828724352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629111.586274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828724352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629111.586284:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629111.586288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629111.586291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629111.586298:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629111.586320:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629111.586331:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629111/real 1778629111] req@ffff954350196d80 x1865013828724480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629111.586344:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828724480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629111.586352:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629111.586354:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629111.586356:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629111.586361:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629111.586378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629111.586384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629111/real 1778629111] req@ffff954350197480 x1865013828724608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629111.586392:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828724608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629111.586407:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629111.586409:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629111.586411:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629111.586428:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629111.586449:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629111.586460:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629111/real 1778629111] req@ffff954350197800 x1865013828724736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629111.586473:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828724736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629111.586482:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629111.586485:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629111.586487:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629111.777756:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629111.777862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629112.545849:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629112.801788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629112.801872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629113.569830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629113.825764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629113.825857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629114.593823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629114.849763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629114.849853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629115.617851:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629115.873772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629115.873881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629116.641819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629116.705841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629116.705855:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629116.705861:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629116.705874:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13451 00000100:00080000:0.0:1778629116.705890:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629116.705898:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629116.705947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828724864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629116.705996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629116.706019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629116.706022:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629116.706029:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13451 00000100:00100000:1.0:1778629116.706029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828724864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629116.706034:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629116.706040:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629116.706052:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629116.706068:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828724992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629116.706080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629116.706084:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629116.706085:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629116.706090:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13451 00000100:00080000:0.0:1778629116.706094:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629116.706098:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629116.706104:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778629116.706104:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629116.706119:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828725120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629116.706130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629116.706134:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629116.706136:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629116.706141:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13451 00000100:00080000:0.0:1778629116.706145:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629116.706150:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629116.706154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629116/real 1778629116] req@ffff954342f4b100 x1865013828724864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778629116.706166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629116.706175:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828724864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629116.706181:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828725248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629116.706188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629116.706196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778629116.706199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629116.706201:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828724864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629116.706205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629116.706208:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629116.706214:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13451 00000100:00080000:0.0:1778629116.706218:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629116.706222:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629116.706227:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629116.706232:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629116.706240:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828725376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629116.706253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629116.706265:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629116/real 1778629116] req@ffff954342f48700 x1865013828724992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629116.706274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828724992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629116.706293:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629116.706295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629116.706298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828724992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629116.706302:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629116.706316:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629116.706322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629116/real 1778629116] req@ffff954342f49c00 x1865013828725120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629116.706329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828725120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629116.706334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629116.706336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629116.706338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629116.706341:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629116.706353:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629116.706358:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629116/real 1778629116] req@ffff954342f4b480 x1865013828725248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629116.706365:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828725248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629116.706370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629116.706372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629116.706378:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629116.706391:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629116.706404:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629116.706410:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629116/real 1778629116] req@ffff954342f4a300 x1865013828725376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629116.706417:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828725376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629116.706423:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629116.706425:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629116.706427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629116.897819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629116.897917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629117.665775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629117.921747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629117.921885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629118.689788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629118.945794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629118.945888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629119.713776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629119.969804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629119.969893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629120.737809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629120.993820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629120.993916:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629121.761803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629121.825835:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629121.825850:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629121.825857:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629121.825885:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13456 00000100:00080000:1.0:1778629121.825902:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629121.825911:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629121.825972:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828725504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629121.826007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629121.826032:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629121.826035:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1778629121.826041:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828725504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629121.826042:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13456 00000100:00080000:1.0:1778629121.826052:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629121.826059:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629121.826071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629121.826087:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828725632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629121.826100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629121.826104:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629121.826106:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629121.826110:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13456 00000100:00080000:1.0:1778629121.826117:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629121.826120:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629121.826125:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1778629121.826133:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629121.826138:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828725760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629121.826148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629121.826152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629121.826155:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629121.826160:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13456 00000100:00080000:1.0:1778629121.826164:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629121.826169:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629121.826181:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629121.826183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629121/real 1778629121] req@ffff954350194700 x1865013828725504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629121.826198:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828725888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1778629121.826205:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828725504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629121.826218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629121.826227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629121.826231:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629121.826235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629121.826238:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13456 00000100:00080000:0.0:1778629121.826242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629121.826246:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1778629121.826247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828725504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629121.826256:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629121.826262:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629121.826276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629121.826278:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828726016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1778629121.826312:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629121.826325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629121/real 1778629121] req@ffff954350197b80 x1865013828725632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629121.826336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828725632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629121.826343:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629121.826346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629121.826348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629121.826352:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:0.0:1778629121.826363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629121.826369:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629121/real 1778629121] req@ffff954350196a00 x1865013828725760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629121.826377:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828725760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629121.826382:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629121.826384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629121.826386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629121.826390:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629121.826402:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629121.826407:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629121/real 1778629121] req@ffff954350194e00 x1865013828725888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629121.826425:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828725888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629121.826431:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629121.826433:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629121.826435:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828725888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629121.826446:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:0.0:1778629121.826458:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629121.826464:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629121/real 1778629121] req@ffff954350194a80 x1865013828726016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629121.826472:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828726016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629121.826477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629121.826479:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:0.0:1778629121.826481:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629122.017806:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629122.017902:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629122.785807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629123.041776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629123.041826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629123.809780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629124.065799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629124.065897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629124.833830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629125.089822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629125.089906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629125.857732:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629126.113851:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629126.113914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629126.881760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629126.945848:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629126.945856:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629126.945860:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629126.945869:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13461 00000100:00080000:0.0:1778629126.945880:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629126.945885:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629126.945932:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828726144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629126.945967:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629126.945972:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629126.945974:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629126.945979:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13461 00000100:00080000:0.0:1778629126.945985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629126.945990:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629126.946000:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629126.946016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828726272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629126.946030:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828726144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629126.946036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629126.946048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629126.946051:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629126.946056:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13461 00000100:00080000:0.0:1778629126.946061:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629126.946066:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629126.946072:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629126.946088:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828726400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629126.946099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629126.946103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629126.946106:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629126.946119:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13461 00000800:00000100:1.0:1778629126.946121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629126.946132:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629126.946140:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629126.946149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629126.946168:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828726528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629126.946173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629126/real 1778629126] req@ffff954342f49880 x1865013828726144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629126.946223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629126.946227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778629126.946231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828726144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629126.946232:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629126.946248:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13461 00000100:00080000:1.0:1778629126.946250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629126.946255:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629126.946256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778629126.946263:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629126.946269:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629126.946274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828726144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629126.946290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828726656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629126.946314:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629126.946338:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629126.946348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629126/real 1778629126] req@ffff954342f4bb80 x1865013828726272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629126.946357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828726272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629126.946363:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629126.946366:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629126.946368:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629126.946372:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629126.946383:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629126.946388:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629126/real 1778629126] req@ffff954342f48e00 x1865013828726400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629126.946396:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828726400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629126.946401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629126.946403:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629126.946405:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629126.946409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629126.946420:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629126.946434:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629126/real 1778629126] req@ffff954342f4aa00 x1865013828726528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629126.946441:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828726528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629126.946447:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629126.946449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629126.946451:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629126.946456:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629126.946467:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629126.946472:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629126/real 1778629126] req@ffff954342f49180 x1865013828726656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629126.946480:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828726656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629126.946486:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629126.946487:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629126.946489:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629127.137834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629127.137880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629127.905740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629128.161825:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629128.161931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629128.929763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629129.185776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629129.185888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629129.953766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629130.209799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629130.209891:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629130.977789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629131.233775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629131.233877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629132.001779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629132.065797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629132.065805:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629132.065809:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629132.065817:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13466 00000100:00080000:0.0:1778629132.065828:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629132.065834:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629132.065870:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828726784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629132.065902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629132.065921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629132.065925:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629132.065931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13466 00000100:00080000:0.0:1778629132.065937:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629132.065944:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629132.065945:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828726784:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778629132.065955:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629132.065971:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828726912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629132.065988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629132.065991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629132.065992:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629132.065995:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13466 00000100:00080000:0.0:1778629132.066001:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629132.066004:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629132.066007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778629132.066011:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629132.066019:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828727040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629132.066027:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629132.066030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629132.066031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629132.066034:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13466 00000100:00080000:0.0:1778629132.066037:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629132.066040:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629132.066046:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629132/real 1778629132] req@ffff954342f48380 x1865013828726784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778629132.066050:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629132.066068:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828726784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629132.066071:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828727168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629132.066090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629132.066099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629132.066100:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629132.066103:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13466 00000100:00080000:0.0:1778629132.066108:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629132.066111:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1778629132.066113:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629132.066118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778629132.066119:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629132.066125:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828726784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629132.066130:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828727296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629132.066151:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629132.066191:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629132.066206:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629132/real 1778629132] req@ffff954342f48000 x1865013828726912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629132.066218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828726912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629132.066224:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629132.066228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629132.066231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828726912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629132.066236:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629132.066253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629132.066260:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629132/real 1778629132] req@ffff954342f49180 x1865013828727040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629132.066268:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828727040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629132.066273:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629132.066275:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629132.066277:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629132.066287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629132.066301:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629132.066307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629132/real 1778629132] req@ffff954342f4aa00 x1865013828727168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629132.066315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828727168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629132.066320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629132.066322:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629132.066323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629132.066327:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629132.066336:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629132.066341:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629132/real 1778629132] req@ffff954342f48e00 x1865013828727296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629132.066355:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828727296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629132.066360:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629132.066362:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629132.066364:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629132.257745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629132.257844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629133.025791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778629133.282295:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629133.282356:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629134.049810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629134.305897:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629134.306147:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629135.073846:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629135.329952:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629135.330034:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629136.097786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629136.353865:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629136.354197:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629137.121809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629137.185788:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629137.185827:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629137.185833:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629137.185846:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13471 00000100:00080000:1.0:1778629137.185861:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629137.185870:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629137.185916:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828727424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629137.185956:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629137.185960:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629137.185962:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629137.185967:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13471 00000100:00080000:1.0:1778629137.185972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629137.185976:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629137.185987:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629137.186002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828727552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629137.186014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629137.186017:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828727424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629137.186019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629137.186023:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629137.186029:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13471 00000100:00080000:1.0:1778629137.186033:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629137.186037:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629137.186042:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629137.186055:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828727680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629137.186072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629137.186076:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629137.186078:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629137.186083:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13471 00000100:00080000:1.0:1778629137.186089:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629137.186093:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629137.186100:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778629137.186101:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629137.186117:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828727808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629137.186129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629137.186133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629137.186135:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629137.186140:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13471 00000100:00100000:2.0:1778629137.186143:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629137/real 1778629137] req@ffff954350195c00 x1865013828727424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629137.186146:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629137.186156:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629137.186162:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629137.186165:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828727424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629137.186174:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778629137.186180:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828727936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629137.186180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629137.186186:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828727424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629137.186216:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629137.186234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629137.186245:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629137/real 1778629137] req@ffff954350195500 x1865013828727552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629137.186258:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828727552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629137.186267:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629137.186270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629137.186282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629137.186290:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629137.186313:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629137.186321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629137/real 1778629137] req@ffff954350195f80 x1865013828727680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629137.186330:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828727680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629137.186335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629137.186337:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629137.186340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629137.186346:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629137.186365:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629137.186376:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629137/real 1778629137] req@ffff954350194380 x1865013828727808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629137.186389:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828727808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629137.186398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629137.186404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629137.186408:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629137.186414:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629137.186432:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629137.186457:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629137/real 1778629137] req@ffff954350195180 x1865013828727936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629137.186475:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828727936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629137.186485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629137.186489:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629137.186492:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828727936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629137.377850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629137.378142:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629138.145795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629138.402495:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629138.402602:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629139.169822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629139.425810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629139.425895:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629140.193836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629140.449818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629140.449893:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629141.217801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629141.473798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629141.473833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629142.241802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629142.305843:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629142.305858:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629142.305864:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629142.305878:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13476 00000100:00080000:0.0:1778629142.305894:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629142.305902:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629142.305958:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828728064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629142.306021:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629142.306025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629142.306027:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629142.306033:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13476 00000100:00080000:0.0:1778629142.306038:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629142.306042:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629142.306054:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629142.306073:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828728192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629142.306084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828728064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629142.306089:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629142.306095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629142.306097:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629142.306102:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13476 00000100:00080000:0.0:1778629142.306105:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629142.306109:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629142.306115:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629142.306129:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828728320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629142.306141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629142.306144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629142.306146:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629142.306151:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13476 00000100:00080000:0.0:1778629142.306164:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778629142.306169:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629142.306172:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629142.306180:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629142.306196:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828728448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629142.306208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629142.306212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629142.306214:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629142.306219:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13476 00000100:00100000:2.0:1778629142.306223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629142/real 1778629142] req@ffff954342f4bb80 x1865013828728064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629142.306225:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629142.306230:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629142.306235:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629142.306252:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828728576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629142.306266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828728064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629142.306279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629142.306295:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629142.306299:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828728064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629142.306330:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629142.306355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629142.306368:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629142/real 1778629142] req@ffff954342f49880 x1865013828728192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629142.306381:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828728192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629142.306390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629142.306393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629142.306397:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629142.306403:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629142.306422:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629142.306432:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629142/real 1778629142] req@ffff954342f4a300 x1865013828728320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629142.306445:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828728320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629142.306453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629142.306457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629142.306460:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629142.306482:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629142.306512:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629142.306524:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629142/real 1778629142] req@ffff954342f4b480 x1865013828728448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629142.306536:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828728448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629142.306544:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629142.306548:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629142.306551:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629142.306557:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629142.306577:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629142.306586:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629142/real 1778629142] req@ffff954342f49c00 x1865013828728576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629142.306598:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828728576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629142.306606:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629142.306610:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629142.306613:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629142.497805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629142.497867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629143.265769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629143.521832:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629143.521915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629144.289779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629144.545781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629144.545864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629145.313771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629145.569826:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629145.569895:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629146.337779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629146.593803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629146.593880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629147.361845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629147.425803:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629147.425815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629147.425839:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629147.425849:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13481 00000100:00080000:1.0:1778629147.425861:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629147.425868:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629147.425906:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828728704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629147.425934:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629147.425938:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629147.425940:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629147.425945:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13481 00000100:00080000:1.0:1778629147.425950:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629147.425955:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629147.425967:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629147.425968:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828728704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629147.425987:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828728832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629147.425996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629147.425998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629147.426000:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629147.426003:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13481 00000100:00080000:1.0:1778629147.426006:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629147.426009:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629147.426013:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629147.426022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828728960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629147.426039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629147.426041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629147.426043:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629147.426046:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13481 00000100:00080000:1.0:1778629147.426049:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629147.426052:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629147.426057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778629147.426058:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629147.426070:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828729088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629147.426078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629147.426081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629147.426082:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629147.426084:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13481 00000100:00080000:1.0:1778629147.426087:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629147.426089:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629147.426092:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629147.426101:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828729216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629147.426105:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629147/real 1778629147] req@ffff954350195880 x1865013828728704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629147.426128:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828728704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629147.426138:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629147.426142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629147.426145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828728704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629147.426168:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629147.426185:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629147.426193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629147/real 1778629147] req@ffff954350195180 x1865013828728832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629147.426202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828728832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629147.426208:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629147.426210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629147.426212:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629147.426227:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629147.426242:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629147.426248:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629147/real 1778629147] req@ffff954350194380 x1865013828728960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629147.426256:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828728960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629147.426261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629147.426264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629147.426266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828728960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629147.426270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629147.426280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629147.426285:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629147/real 1778629147] req@ffff954350195f80 x1865013828729088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629147.426292:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828729088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629147.426298:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629147.426300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629147.426302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629147.426306:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629147.426315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629147.426320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629147/real 1778629147] req@ffff954350195500 x1865013828729216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629147.426333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828729216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629147.426340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629147.426343:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629147.426346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629147.617829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629147.617918:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629148.385806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629148.641795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629148.641855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629149.409771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629149.665788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629149.665872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629150.433785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629150.689757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629150.689841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629151.457793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629151.713760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629151.713832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629152.481849:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629152.545783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629152.545792:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629152.545797:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629152.545804:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13487 00000100:00080000:1.0:1778629152.545831:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629152.545837:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629152.545866:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828729344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629152.545889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629152.545892:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629152.545893:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629152.545896:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13487 00000100:00080000:1.0:1778629152.545898:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629152.545900:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629152.545908:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629152.545916:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828729472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629152.545924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629152.545926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629152.545927:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629152.545930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13487 00000100:00080000:1.0:1778629152.545931:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629152.545933:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629152.545937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629152.545943:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828729600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629152.545948:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828729344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629152.545957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629152.545965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629152.545967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629152.545971:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13487 00000100:00080000:1.0:1778629152.545974:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629152.545976:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629152.545980:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629152.545989:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828729728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629152.545996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629152.545998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629152.546000:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629152.546011:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13487 00000100:00080000:1.0:1778629152.546013:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629152.546017:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629152.546020:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778629152.546022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629152.546032:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828729856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629152.546058:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629152/real 1778629152] req@ffff954350195c00 x1865013828729344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629152.546075:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828729344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629152.546087:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629152.546091:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629152.546095:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828729344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629152.546111:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629152.546146:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629152.546160:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629152/real 1778629152] req@ffff954350194a80 x1865013828729472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629152.546173:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828729472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629152.546182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629152.546185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629152.546189:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629152.546223:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629152.546245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629152.546257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629152/real 1778629152] req@ffff954350194e00 x1865013828729600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629152.546285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828729600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629152.546295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629152.546299:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629152.546302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629152.546309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629152.546326:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629152.546334:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629152/real 1778629152] req@ffff954350196a00 x1865013828729728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629152.546348:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828729728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629152.546355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629152.546359:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629152.546362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629152.546368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629152.546389:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629152.546400:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629152/real 1778629152] req@ffff954350197b80 x1865013828729856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629152.546412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828729856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629152.546421:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629152.546423:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629152.546425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828729856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629152.737745:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629152.737939:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629153.505797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629153.761712:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629153.761812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629154.529782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629154.785794:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629154.785949:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629155.553770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629155.809787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629155.809881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629156.577827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629156.833811:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629156.833880:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629157.601824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629157.665779:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629157.665790:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629157.665795:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629157.665807:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13492 00000100:00080000:1.0:1778629157.665821:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629157.665829:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629157.665875:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828729984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629157.665920:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629157.665924:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629157.665926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629157.665930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13492 00000100:00080000:1.0:1778629157.665935:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629157.665940:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629157.665950:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629157.665965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828730112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629157.665965:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828729984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629157.665990:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629157.665993:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629157.665995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629157.665999:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13492 00000100:00080000:1.0:1778629157.666003:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629157.666008:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629157.666014:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629157.666027:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828730240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629157.666036:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629157.666040:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629157.666041:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778629157.666047:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629157.666051:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13492 00000100:00080000:1.0:1778629157.666063:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629157.666069:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629157.666076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629157.666082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629157/real 1778629157] req@ffff954350194700 x1865013828729984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629157.666091:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828730368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629157.666101:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828729984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629157.666104:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629157.666122:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629157.666128:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629157.666131:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629157.666137:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13492 00000100:00080000:2.0:1778629157.666142:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629157.666143:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629157.666153:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629157.666154:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828729984:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778629157.666171:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629157.666189:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828730496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629157.666198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629157.666230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629157.666244:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629157/real 1778629157] req@ffff954350197800 x1865013828730112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629157.666267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828730112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629157.666275:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629157.666278:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629157.666281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629157.666287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629157.666305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629157.666314:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629157/real 1778629157] req@ffff954350197480 x1865013828730240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629157.666338:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828730240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629157.666345:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629157.666348:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629157.666351:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629157.666357:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629157.666375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629157.666385:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629157/real 1778629157] req@ffff954350196d80 x1865013828730368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629157.666394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828730368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629157.666401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629157.666404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629157.666406:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629157.666422:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629157.666438:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629157.666447:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629157/real 1778629157] req@ffff954350194000 x1865013828730496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629157.666456:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828730496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629157.666477:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629157.666481:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629157.666484:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629157.857763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629157.857825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629158.625768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629158.881778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629158.881859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629159.649813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629159.905812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629159.905881:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629160.673793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629160.929771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629160.929834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629161.697825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629161.953784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629161.953838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629162.721816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629162.785756:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629162.785764:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629162.785768:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629162.785776:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13497 00000100:00080000:1.0:1778629162.785785:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629162.785790:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629162.785821:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828730624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629162.785844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629162.785859:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629162.785860:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629162.785863:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13497 00000100:00080000:1.0:1778629162.785866:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629162.785869:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629162.785876:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629162.785886:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828730752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629162.785893:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629162.785897:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629162.785898:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629162.785899:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828730624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629162.785902:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13497 00000100:00080000:1.0:1778629162.785905:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629162.785908:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629162.785911:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629162.785917:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828730880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629162.785922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629162.785924:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629162.785925:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629162.785928:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13497 00000100:00080000:1.0:1778629162.785929:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629162.785932:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629162.785939:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629162.785945:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828731008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629162.785951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629162.785953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629162.785954:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629162.785956:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13497 00000100:00080000:1.0:1778629162.785959:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629162.785962:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629162.785964:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778629162.785966:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629162.785973:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828731136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629162.786001:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629162/real 1778629162] req@ffff954350197100 x1865013828730624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629162.786021:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828730624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629162.786031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629162.786035:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629162.786039:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828730624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629162.786071:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629162.786093:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629162.786121:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629162/real 1778629162] req@ffff954350194000 x1865013828730752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629162.786135:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828730752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629162.786142:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629162.786146:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629162.786149:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629162.786154:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629162.786173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629162.786183:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629162/real 1778629162] req@ffff954350196d80 x1865013828730880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629162.786195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828730880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629162.786202:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629162.786205:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629162.786208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828730880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629162.786214:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629162.786232:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629162.786242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629162/real 1778629162] req@ffff954350197480 x1865013828731008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629162.786253:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828731008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629162.786266:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629162.786269:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629162.786272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629162.786278:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629162.786298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629162.786307:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629162/real 1778629162] req@ffff954350197800 x1865013828731136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629162.786318:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828731136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629162.786325:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629162.786327:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629162.786329:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629162.977830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629162.977844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629163.745800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629164.001792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629164.001869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629164.769755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629165.025789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629165.025877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629165.793772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629166.049774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629166.049867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629166.817743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629167.073808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629167.073883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629167.841804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629167.905802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629167.905810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629167.905814:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629167.905821:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13502 00000100:00080000:0.0:1778629167.905831:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629167.905836:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629167.905870:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828731264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629167.905896:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629167.905914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629167.905917:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629167.905921:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13502 00000100:00080000:0.0:1778629167.905926:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629167.905930:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629167.905938:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828731264:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778629167.905939:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629167.905966:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828731392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629167.905978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629167.905980:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629167.905981:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629167.905984:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13502 00000100:00080000:0.0:1778629167.905987:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629167.905990:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629167.905993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629167.906002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828731520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629167.906008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629167.906011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629167.906012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629167.906014:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13502 00000100:00080000:0.0:1778629167.906016:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629167.906018:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629167.906026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629167.906034:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828731648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629167.906049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629167.906050:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731392:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778629167.906056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629167.906059:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629167.906062:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13502 00000100:00080000:0.0:1778629167.906066:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629167.906069:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629167.906072:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629167.906079:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:0.0:1778629167.906084:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828731776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629167.906107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629167.906130:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629168.097779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629168.097861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629168.865777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629169.121843:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629169.121924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629169.889829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629170.145729:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629170.145818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629170.913772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778629170.977995:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778629170.978041:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778629170.978059:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629170.978107:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629170.978116:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629170.978124:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629170.978131:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778629170.978226:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629167/real 1778629170] req@ffff954342f49500 x1865013828731648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629170.978253:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828731648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629170.978264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629170.978270:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629170.978274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629170.978282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629167/real 1778629170] req@ffff954342f4ad80 x1865013828731776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629170.978294:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828731776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629170.978304:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629170.978323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629170.978327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629170.978334:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629167/real 1778629170] req@ffff954342f4b100 x1865013828731392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629170.978346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828731392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629170.978354:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629170.978357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629170.978360:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629170.978367:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629167/real 1778629170] req@ffff954342f48a80 x1865013828731520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629170.978377:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828731520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629170.978384:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629170.978388:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629170.978391:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828731520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629170.978397:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629167/real 1778629170] req@ffff954342f48700 x1865013828731264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629170.978409:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828731264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629170.978415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629170.978422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629170.978425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828731264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629171.169740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629171.169820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629171.937787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629172.193760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629172.193811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629172.961841:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629173.025831:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629173.025843:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629173.025849:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629173.025874:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13507 00000100:00080000:1.0:1778629173.025887:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629173.025895:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629173.025939:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828731904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629173.025973:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629173.025977:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629173.025979:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629173.025984:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13507 00000100:00080000:1.0:1778629173.026006:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629173.026012:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629173.026016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828731904:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778629173.026024:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629173.026048:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828732032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629173.026060:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629173.026064:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629173.026065:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629173.026069:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13507 00000100:00080000:1.0:1778629173.026078:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629173.026082:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629173.026087:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629173.026098:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828732160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629173.026103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629173.026114:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629173.026117:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629173.026119:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629173.026125:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13507 00000100:00080000:1.0:1778629173.026129:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629173.026134:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629173.026139:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629173.026149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629173/real 1778629173] req@ffff954350194700 x1865013828731904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629173.026151:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828732288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629173.026187:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828731904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629173.026216:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629173.026219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629173.026220:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629173.026224:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629173.026228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629173.026232:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13507 00000100:00100000:2.0:1778629173.026236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828731904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629173.026238:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629173.026247:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629173.026253:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629173.026265:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828732416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629173.026277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629173.026307:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629173.026321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629173/real 1778629173] req@ffff954350197b80 x1865013828732032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629173.026333:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828732032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629173.026342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629173.026346:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629173.026361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629173.026368:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629173.026390:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629173.026400:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629173/real 1778629173] req@ffff954350196a00 x1865013828732160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629173.026410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828732160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629173.026416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629173.026420:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629173.026423:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629173.026429:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629173.026447:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629173.026455:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629173/real 1778629173] req@ffff954350194e00 x1865013828732288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629173.026466:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828732288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629173.026473:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629173.026476:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629173.026479:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629173.026502:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629173.026520:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629173.026541:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629173/real 1778629173] req@ffff954350194a80 x1865013828732416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629173.026552:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828732416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629173.026560:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629173.026563:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629173.026566:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629173.217805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629173.217879:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629173.985784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629174.241773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629174.241846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629175.009789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629175.265777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629175.265829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629176.033836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629176.289801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629176.289866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629177.057843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629177.313807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629177.313908:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629178.081848:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629178.145728:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629178.145736:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629178.145740:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629178.145747:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13512 00000100:00080000:1.0:1778629178.145767:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629178.145773:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629178.145801:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828732544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629178.145822:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629178.145824:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629178.145825:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629178.145828:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13512 00000100:00080000:1.0:1778629178.145830:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629178.145832:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629178.145854:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629178.145862:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828732544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629178.145865:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828732672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629178.145875:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629178.145877:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629178.145878:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629178.145881:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13512 00000100:00080000:1.0:1778629178.145883:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629178.145886:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629178.145888:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629178.145896:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828732800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629178.145901:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629178.145904:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629178.145905:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629178.145907:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13512 00000100:00080000:1.0:1778629178.145909:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629178.145911:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629178.145914:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778629178.145918:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629178.145924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828732928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629178.145930:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629178.145933:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629178.145937:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629178.145941:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13512 00000100:00080000:1.0:1778629178.145944:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629178.145947:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629178/real 1778629178] req@ffff954350195c00 x1865013828732544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629178.145949:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629178.145955:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629178.145966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828733056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629178.145970:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828732544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629178.145982:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629178.145985:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629178.145994:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828732544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629178.146012:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629178.146028:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629178.146035:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629178/real 1778629178] req@ffff954350195500 x1865013828732672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629178.146045:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828732672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629178.146051:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629178.146053:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629178.146054:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629178.146058:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629178.146069:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629178.146074:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629178/real 1778629178] req@ffff954350195f80 x1865013828732800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629178.146080:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828732800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629178.146085:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629178.146086:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629178.146088:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629178.146091:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629178.146100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629178.146112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629178/real 1778629178] req@ffff954350194380 x1865013828732928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629178.146119:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828732928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629178.146128:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629178.146130:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629178.146132:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828732928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629178.146135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629178.146146:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629178.146152:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629178/real 1778629178] req@ffff954350195180 x1865013828733056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629178.146158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828733056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629178.146163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629178.146164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629178.146166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733056:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778629178.337841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629178.337999:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629179.105776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629179.361773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629179.361836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629180.129785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629180.385776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629180.385829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629181.153809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629181.409769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629181.409847:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629182.177796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629182.433809:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629182.433864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629183.201765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629183.265786:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629183.265796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629183.265801:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629183.265810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13517 00000100:00080000:0.0:1778629183.265823:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629183.265829:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629183.265869:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828733184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629183.265910:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629183.265914:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629183.265916:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629183.265921:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13517 00000100:00080000:0.0:1778629183.265944:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629183.265948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629183.265957:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629183.265969:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828733184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629183.265980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828733312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629183.265989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629183.265991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629183.265993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629183.265995:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13517 00000100:00080000:0.0:1778629183.265997:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629183.265999:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629183.266002:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629183.266011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828733440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629183.266017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629183.266019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629183.266020:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629183.266023:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13517 00000100:00080000:0.0:1778629183.266024:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629183.266026:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629183.266029:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629183.266035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828733568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629183.266050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778629183.266051:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629183.266054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629183.266056:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629183.266059:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13517 00000100:00080000:0.0:1778629183.266062:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629183.266064:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629183.266068:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629183.266076:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828733696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629183.266110:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629183/real 1778629183] req@ffff954342f4a680 x1865013828733184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629183.266132:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828733184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629183.266143:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629183.266147:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629183.266152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828733184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629183.266180:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629183.266200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629183.266211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629183/real 1778629183] req@ffff954342f49f80 x1865013828733312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629183.266222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828733312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629183.266230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629183.266233:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629183.266236:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629183.266242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629183.266261:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629183.266278:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629183/real 1778629183] req@ffff954342f48700 x1865013828733440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629183.266290:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828733440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629183.266297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629183.266300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629183.266303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629183.266309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629183.266330:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629183.266339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629183/real 1778629183] req@ffff954342f48a80 x1865013828733568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629183.266351:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828733568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629183.266358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629183.266361:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629183.266364:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629183.266369:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629183.266386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629183.266396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629183/real 1778629183] req@ffff954342f4b100 x1865013828733696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629183.266407:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828733696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629183.266425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629183.266428:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629183.266431:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629183.457812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629183.457874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629184.225774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629184.481886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629184.481950:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629185.249772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629185.505797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629185.505872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629186.273737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629186.529822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629186.530022:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629187.297830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629187.553777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629187.553897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629188.321854:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629188.385869:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629188.385892:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629188.385898:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629188.385912:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13522 00000100:00080000:0.0:1778629188.385929:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629188.385936:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629188.385990:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828733824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629188.386026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629188.386031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629188.386037:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629188.386042:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13522 00000100:00080000:0.0:1778629188.386046:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629188.386050:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629188.386090:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828733824:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778629188.386113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629188.386131:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828733952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629188.386144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629188.386149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629188.386151:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629188.386155:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13522 00000100:00080000:0.0:1778629188.386168:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629188.386173:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629188.386177:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778629188.386183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:0.0:1778629188.386191:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828734080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629188.386201:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629188.386206:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629188.386208:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629188.386213:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13522 00000100:00080000:0.0:1778629188.386216:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629188.386220:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629188.386226:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629188.386236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629188/real 1778629188] req@ffff954342f4ad80 x1865013828733824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629188.386240:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828734208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629188.386265:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629188.386268:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629188.386276:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629188.386282:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13522 00000100:00100000:2.0:1778629188.386285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828733824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629188.386288:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629188.386297:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778629188.386301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778629188.386304:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778629188.386309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629188.386314:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828733824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629188.386321:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828734336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629188.386348:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629188.386375:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629188.386397:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629188/real 1778629188] req@ffff954342f49500 x1865013828733952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629188.386411:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828733952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629188.386420:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629188.386424:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629188.386427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828733952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629188.386433:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629188.386455:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629188.386474:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629188/real 1778629188] req@ffff954342f49c00 x1865013828734080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629188.386487:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828734080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629188.386494:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629188.386497:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629188.386500:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629188.386506:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629188.386527:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629188.386544:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629188/real 1778629188] req@ffff954342f4b480 x1865013828734208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629188.386565:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828734208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629188.386579:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629188.386583:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629188.386586:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629188.386592:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629188.386612:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629188.386623:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629188/real 1778629188] req@ffff954342f4a300 x1865013828734336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629188.386634:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828734336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629188.386643:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629188.386646:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629188.386649:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629188.577834:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629188.577911:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629189.345794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629189.601835:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629189.601914:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629190.369775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629190.625758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629190.625837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629191.393774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629191.649789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629191.649980:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629192.417777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629192.673785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629192.673831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629193.441800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629193.505803:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629193.505815:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629193.505821:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629193.505833:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13528 00000100:00080000:3.0:1778629193.505848:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629193.505855:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629193.505902:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828734464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629193.505937:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629193.505940:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629193.505942:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629193.505947:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13528 00000100:00080000:3.0:1778629193.505950:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629193.505954:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629193.505965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629193.505978:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828734592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629193.505983:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828734464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629193.505994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629193.505997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629193.505999:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629193.506002:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13528 00000100:00080000:3.0:1778629193.506005:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629193.506009:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629193.506021:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629193.506034:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828734720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629193.506043:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629193.506047:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629193.506049:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629193.506053:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13528 00000100:00080000:3.0:1778629193.506067:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629193.506071:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629193.506076:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629193.506085:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828734848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629193.506090:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629193.506100:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629193.506104:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629193.506105:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629193.506110:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13528 00000100:00080000:3.0:1778629193.506114:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629193.506118:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629193.506122:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629193.506131:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828734976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629193.506140:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629193/real 1778629193] req@ffff95434701c700 x1865013828734464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629193.506163:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828734464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629193.506175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629193.506180:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629193.506183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828734464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629193.506206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629193.506224:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629193.506243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629193/real 1778629193] req@ffff95434701c000 x1865013828734592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629193.506255:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828734592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629193.506263:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629193.506266:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629193.506269:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629193.506275:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629193.506295:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629193.506303:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629193/real 1778629193] req@ffff95434701ea00 x1865013828734720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629193.506328:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828734720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629193.506336:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629193.506340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629193.506343:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629193.506349:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629193.506369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629193.506377:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629193/real 1778629193] req@ffff95434701d180 x1865013828734848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629193.506385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828734848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629193.506390:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629193.506392:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629193.506393:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629193.506397:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629193.506407:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629193.506411:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629193/real 1778629193] req@ffff95434701ca80 x1865013828734976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629193.506418:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828734976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629193.506438:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629193.506440:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629193.506442:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828734976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629193.697804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629193.697858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629194.465775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629194.721804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629194.721875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629195.489776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629195.745758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629195.745831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629196.513801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629196.769758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629196.769820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629197.537813:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629197.793751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629197.793985:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629198.561781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629198.625845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629198.625858:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629198.625863:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629198.625875:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13533 00000100:00080000:3.0:1778629198.625891:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629198.625899:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629198.625944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828735104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629198.625972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629198.625975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629198.625976:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629198.625980:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13533 00000100:00080000:3.0:1778629198.625982:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629198.625985:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629198.626000:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629198.626018:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828735232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629198.626029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629198.626033:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828735104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629198.626034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629198.626037:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629198.626046:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13533 00000100:00080000:3.0:1778629198.626050:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629198.626072:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629198.626079:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629198.626092:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828735360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629198.626102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629198.626105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629198.626107:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629198.626111:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13533 00000800:00000100:2.0:1778629198.626116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629198.626118:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629198.626123:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629198.626141:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629198.626154:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828735488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629198.626162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629198.626164:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629198/real 1778629198] req@ffff95434701df80 x1865013828735104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629198.626165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629198.626169:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629198.626179:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13533 00000100:00080000:3.0:1778629198.626184:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629198.626189:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629198.626193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828735104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778629198.626196:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778629198.626210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778629198.626215:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828735616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629198.626217:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629198.626224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828735104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629198.626258:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629198.626282:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629198.626301:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629198/real 1778629198] req@ffff95434701e680 x1865013828735232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629198.626313:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828735232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629198.626321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629198.626324:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629198.626327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629198.626332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629198.626365:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629198.626378:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629198/real 1778629198] req@ffff95434701f480 x1865013828735360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629198.626391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828735360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629198.626398:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629198.626401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629198.626404:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629198.626410:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629198.626428:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629198.626437:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629198/real 1778629198] req@ffff95434701ca80 x1865013828735488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629198.626446:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828735488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629198.626461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629198.626464:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629198.626467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629198.626472:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629198.626490:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629198.626498:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629198/real 1778629198] req@ffff95434701d180 x1865013828735616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629198.626509:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828735616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629198.626518:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629198.626521:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629198.626524:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629198.817798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629198.817871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629199.585773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629199.842613:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629199.843316:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629200.609746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629200.866208:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629200.866378:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629201.633768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629201.890194:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629201.891208:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629202.657827:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629202.914398:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629202.914424:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629203.681742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629203.745770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629203.745777:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629203.745780:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629203.745787:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13538 00000100:00080000:3.0:1778629203.745797:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629203.745801:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629203.745831:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828735744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629203.745852:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629203.745854:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629203.745855:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629203.745858:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13538 00000100:00080000:3.0:1778629203.745862:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629203.745865:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629203.745874:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629203.745887:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828735872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629203.745892:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828735744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629203.745903:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629203.745906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629203.745907:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629203.745910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13538 00000100:00080000:3.0:1778629203.745933:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629203.745937:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629203.745941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778629203.745945:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778629203.745951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828736000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629203.745958:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629203.745960:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629203.745961:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629203.745964:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13538 00000100:00080000:3.0:1778629203.745967:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629203.745970:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629203.745973:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629203.745976:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629203/real 1778629203] req@ffff95434701ea00 x1865013828735744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629203.745981:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828736128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629203.746005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629203.746009:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828735744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629203.746009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629203.746013:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629203.746020:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13538 00000100:00080000:2.0:1778629203.746021:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778629203.746022:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629203.746027:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778629203.746027:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629203.746031:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629203.746038:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828735744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778629203.746043:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828736256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629203.746062:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629203.746080:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629203.746090:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629203/real 1778629203] req@ffff95434701c000 x1865013828735872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629203.746098:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828735872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629203.746103:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629203.746105:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629203.746112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828735872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629203.746116:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629203.746128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629203.746133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629203/real 1778629203] req@ffff95434701c700 x1865013828736000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629203.746140:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828736000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629203.746152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629203.746154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629203.746156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629203.746159:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629203.746171:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629203.746176:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629203/real 1778629203] req@ffff95434701c380 x1865013828736128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629203.746182:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828736128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629203.746187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629203.746188:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629203.746190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629203.746198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629203.746207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629203.746211:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629203/real 1778629203] req@ffff95434701d500 x1865013828736256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629203.746218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828736256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629203.746222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629203.746223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629203.746228:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736256:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778629203.937889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629203.938213:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629204.705782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629204.961909:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629204.961964:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629205.729741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629205.985866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629205.985992:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629206.753763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629207.009729:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629207.009777:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629207.777782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629208.033783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629208.033849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629208.801800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629208.865770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629208.865777:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629208.865781:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629208.865789:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13543 00000100:00080000:3.0:1778629208.865798:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629208.865804:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629208.865833:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828736384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629208.865855:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629208.865859:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629208.865861:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629208.865864:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13543 00000100:00080000:3.0:1778629208.865868:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629208.865871:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629208.865879:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629208.865892:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828736512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629208.865902:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629208.865906:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828736384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629208.865907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629208.865922:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629208.865927:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13543 00000100:00080000:3.0:1778629208.865932:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629208.865936:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629208.865941:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629208.865950:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828736640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629208.865957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629208.865959:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629208.865960:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629208.865962:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13543 00000100:00080000:3.0:1778629208.865965:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629208.865968:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629208.865972:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778629208.865974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629208.865983:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828736768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629208.865989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629208.865992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629208.865993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629208.865995:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13543 00000100:00080000:3.0:1778629208.865997:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629208.866000:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629208.866002:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629208.866011:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629208/real 1778629208] req@ffff95434701f800 x1865013828736384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629208.866018:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828736896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629208.866042:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828736384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629208.866053:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629208.866057:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629208.866062:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828736384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629208.866085:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629208.866107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629208.866119:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629208/real 1778629208] req@ffff95434701d880 x1865013828736512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629208.866131:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828736512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629208.866145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629208.866157:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629208.866161:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629208.866167:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629208.866187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629208.866198:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629208/real 1778629208] req@ffff95434701f100 x1865013828736640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629208.866210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828736640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629208.866218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629208.866221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629208.866224:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629208.866230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629208.866250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629208.866259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629208/real 1778629208] req@ffff95434701d500 x1865013828736768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629208.866270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828736768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629208.866277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629208.866280:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629208.866289:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629208.866295:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629208.866314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629208.866323:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629208/real 1778629208] req@ffff95434701c380 x1865013828736896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629208.866334:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828736896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629208.866342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629208.866345:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629208.866348:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828736896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629209.057783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629209.057844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629209.825763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629210.081736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629210.081804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629210.849775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629211.105759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629211.105825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629211.873816:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629212.129871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629212.129873:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629212.897826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629213.153850:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629213.153867:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629213.921815:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629213.985726:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629213.985736:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629213.985741:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629213.985750:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13548 00000100:00080000:3.0:1778629213.985761:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629213.985766:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629213.985806:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828737024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629213.985832:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629213.985835:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629213.985836:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629213.985839:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13548 00000100:00080000:3.0:1778629213.985842:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629213.985845:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629213.985852:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629213.985861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828737152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629213.985867:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629213.985881:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629213.985883:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629213.985886:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13548 00000100:00080000:3.0:1778629213.985889:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629213.985894:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629213.985897:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629213.985904:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828737280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629213.985911:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629213.985920:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629213.985924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629213.985926:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828737024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629213.985928:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13548 00000100:00080000:3.0:1778629213.985936:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629213.985947:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629213.985950:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629213.985958:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828737408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629213.985963:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629213.985966:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629213.985967:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629213.985969:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13548 00000100:00080000:3.0:1778629213.985971:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629213.985974:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629213.985983:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629213.986000:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828737536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629213.986032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629213.986091:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629213/real 1778629213] req@ffff95434701c700 x1865013828737024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629213.986139:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828737024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629213.986150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629213.986154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629213.986158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828737024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629213.986167:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629213.986191:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629213.986202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629213/real 1778629213] req@ffff95434701c000 x1865013828737152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629213.986214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828737152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629213.986222:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629213.986226:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629213.986229:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629213.986257:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629213.986276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629213.986285:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629213/real 1778629213] req@ffff95434701ea00 x1865013828737280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629213.986297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828737280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629213.986317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629213.986321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629213.986325:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629213.986331:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629213.986351:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629213.986362:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629213/real 1778629213] req@ffff95434701d180 x1865013828737408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629213.986373:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828737408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629213.986380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629213.986383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629213.986386:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629213.986393:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629213.986412:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629213.986422:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629213/real 1778629213] req@ffff95434701ca80 x1865013828737536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629213.986433:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828737536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629213.986441:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629213.986444:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629213.986452:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629214.177799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629214.177857:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629214.945843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629215.201798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629215.201830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629215.969788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629216.225767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629216.225792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629216.993754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629217.249752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629217.249784:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629218.017770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629218.273798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629218.273861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629219.041790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629219.105906:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629219.105920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629219.105926:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629219.105939:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13553 00000100:00080000:1.0:1778629219.105964:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629219.105972:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629219.106020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828737664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629219.106052:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629219.106056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629219.106058:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629219.106062:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13553 00000100:00080000:1.0:1778629219.106067:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629219.106072:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629219.106083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629219.106124:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828737664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629219.106125:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828737792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629219.106146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629219.106151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629219.106153:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629219.106158:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13553 00000100:00080000:1.0:1778629219.106164:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629219.106169:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629219.106177:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629219.106192:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828737920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629219.106200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629219.106212:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629219.106216:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629219.106218:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629219.106224:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13553 00000100:00080000:1.0:1778629219.106229:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629219.106233:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629219.106239:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629219.106249:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629219/real 1778629219] req@ffff954350195880 x1865013828737664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629219.106256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828738048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629219.106275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629219.106280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828737664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629219.106281:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629219.106284:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629219.106295:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629219.106297:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13553 00000100:00080000:2.0:1778629219.106305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629219.106306:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629219.106316:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629219.106316:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828737664:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778629219.106327:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629219.106342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828738176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629219.106348:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629219.106385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629219.106396:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629219/real 1778629219] req@ffff954350195180 x1865013828737792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629219.106420:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828737792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629219.106426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629219.106428:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629219.106430:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629219.106434:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629219.106447:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629219.106452:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629219/real 1778629219] req@ffff954350194380 x1865013828737920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629219.106459:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828737920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629219.106463:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629219.106465:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629219.106467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828737920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629219.106470:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629219.106486:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629219.106495:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629219/real 1778629219] req@ffff954350195f80 x1865013828738048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629219.106505:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828738048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629219.106523:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629219.106527:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629219.106530:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629219.106547:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629219.106561:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629219.106568:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629219/real 1778629219] req@ffff954350195500 x1865013828738176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629219.106574:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828738176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629219.106580:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629219.106582:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629219.106584:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629219.297773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629219.297862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629220.065804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629220.321813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629220.321872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629221.089817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629221.345819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629221.345883:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629222.113794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629222.369805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629222.369862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629223.137751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629223.393747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629223.393838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629224.161781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629224.225799:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629224.225809:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629224.225827:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629224.225834:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13558 00000100:00080000:2.0:1778629224.225844:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629224.225862:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629224.225896:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828738304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629224.225922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629224.225925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629224.225927:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629224.225930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13558 00000100:00080000:2.0:1778629224.225934:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629224.225938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629224.225947:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629224.225957:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828738304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629224.225961:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828738432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629224.225973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629224.225975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629224.225977:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629224.225980:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13558 00000100:00080000:2.0:1778629224.225983:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629224.225986:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629224.225990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629224.225999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828738560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629224.226018:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778629224.226022:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629224.226024:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629224.226027:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629224.226032:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13558 00000100:00080000:2.0:1778629224.226036:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629224.226041:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629224.226046:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629224.226057:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828738688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629224.226063:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629224/real 1778629224] req@ffff95435878a300 x1865013828738304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629224.226069:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629224.226076:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629224.226079:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629224.226084:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13558 00000100:00080000:2.0:1778629224.226087:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629224.226090:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629224.226093:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828738304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1778629224.226096:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778629224.226105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629224.226108:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629224.226112:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828738304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629224.226118:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828738816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629224.226138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629224.226162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629224.226174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629224/real 1778629224] req@ffff95435878b800 x1865013828738432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629224.226198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828738432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629224.226207:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629224.226210:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629224.226213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629224.226219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629224.226234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629224.226242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629224/real 1778629224] req@ffff954358789500 x1865013828738560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629224.226253:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828738560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629224.226259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629224.226262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629224.226265:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629224.226271:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629224.226289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629224.226299:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629224/real 1778629224] req@ffff954358789880 x1865013828738688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629224.226306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828738688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629224.226319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629224.226321:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629224.226323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629224.226327:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629224.226342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629224.226351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629224/real 1778629224] req@ffff954358788700 x1865013828738816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629224.226360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828738816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629224.226368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629224.226372:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629224.226374:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828738816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629224.417757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629224.417813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629225.185736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629225.441771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629225.441851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629226.209782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629226.465785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629226.465849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629227.233796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629227.489751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629227.489813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629228.257840:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629228.513735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629228.513816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629229.281819:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629229.345854:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629229.345863:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629229.345866:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629229.345876:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13563 00000100:00080000:1.0:1778629229.345886:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629229.345892:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629229.345939:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828738944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629229.345970:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629229.345974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629229.345976:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629229.345980:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13563 00000100:00080000:1.0:1778629229.345985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629229.345989:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629229.345999:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629229.346011:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828739072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629229.346016:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828738944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629229.346027:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629229.346031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629229.346033:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629229.346037:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13563 00000100:00080000:1.0:1778629229.346041:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629229.346044:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629229.346049:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629229.346061:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828739200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629229.346070:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629229.346073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629229.346075:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629229.346086:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13563 00000800:00000100:2.0:1778629229.346091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629229.346093:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629229.346099:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629229.346106:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629229.346120:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828739328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629229.346129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629229/real 1778629229] req@ffff954350195c00 x1865013828738944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629229.346131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629229.346139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629229.346140:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629229.346144:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13563 00000100:00080000:1.0:1778629229.346148:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629229.346154:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629229.346161:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629229.346173:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828739456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629229.346176:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828738944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629229.346199:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629229.346203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629229.346207:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828738944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629229.346230:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629229.346252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629229.346264:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629229/real 1778629229] req@ffff954350194a80 x1865013828739072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629229.346275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828739072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629229.346282:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629229.346286:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629229.346288:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629229.346303:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629229.346325:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629229.346335:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629229/real 1778629229] req@ffff954350194e00 x1865013828739200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629229.346346:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828739200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629229.346353:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629229.346356:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629229.346359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629229.346365:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629229.346384:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629229.346395:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629229/real 1778629229] req@ffff954350196a00 x1865013828739328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629229.346406:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828739328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629229.346413:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629229.346415:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629229.346418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629229.346424:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629229.346440:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629229.346449:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629229/real 1778629229] req@ffff954350197b80 x1865013828739456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629229.346466:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828739456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629229.346483:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629229.346486:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629229.346492:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629229.537768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629229.537828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629230.305807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629230.561797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629230.561882:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629231.329839:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629231.585770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629231.585876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629232.353794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629232.609780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629232.609836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629233.377804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629233.633766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629233.633814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629234.401737:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629234.465764:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629234.465774:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629234.465778:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629234.465787:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13568 00000100:00080000:3.0:1778629234.465796:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629234.465801:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629234.465832:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828739584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629234.465853:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629234.465856:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629234.465857:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629234.465860:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13568 00000100:00080000:3.0:1778629234.465863:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629234.465866:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629234.465876:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629234.465888:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828739712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629234.465894:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828739584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629234.465903:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629234.465906:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629234.465908:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629234.465912:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13568 00000100:00080000:3.0:1778629234.465938:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629234.465941:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629234.465945:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629234.465956:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828739840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629234.465965:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629234.465969:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629234.465970:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629234.465974:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13568 00000100:00080000:3.0:1778629234.465978:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629234.465982:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629234.465985:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739712:192.168.204.102@tcp:38:kworker.0 00000008:00100000:3.0:1778629234.465989:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629234.466005:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828739968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629234.466009:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739840:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778629234.466022:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629234.466026:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629234.466028:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629234.466029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739968:192.168.204.102@tcp:8:kworker.0 00000100:00080000:3.0:1778629234.466034:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13568 00000100:00080000:3.0:1778629234.466039:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629234.466042:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629234.466047:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629234.466059:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828740096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629234.466101:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629234.657779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629234.657844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629235.425766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629235.681827:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629235.681904:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629236.449782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629236.705816:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629236.705886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629237.473804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1778629237.537930:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:0.0:1778629237.537972:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1778629237.537990:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629237.538038:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629237.538047:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629237.538055:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629237.538061:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778629237.538129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629234/real 1778629237] req@ffff95434701d180 x1865013828740096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629237.538155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828740096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629237.538166:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629237.538171:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629237.538175:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629237.538181:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629234/real 1778629237] req@ffff95434701ca80 x1865013828739968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629237.538193:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828739968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629237.538200:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629237.538203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629237.538206:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629237.538213:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629234/real 1778629237] req@ffff95434701e680 x1865013828739712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629237.538225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828739712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629237.538232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629237.538249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629237.538252:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629237.538258:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629234/real 1778629237] req@ffff95434701df80 x1865013828739840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629237.538270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828739840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629237.538276:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629237.538279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629237.538281:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828739840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629237.538287:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629234/real 1778629237] req@ffff95434701f480 x1865013828739584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629237.538297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828739584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629237.538303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629237.538305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629237.538308:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828739584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629237.729765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629237.729820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629238.497765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629238.753764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629238.753824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629239.521745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629239.585768:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629239.585775:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629239.585778:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629239.585788:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13574 00000100:00080000:3.0:1778629239.585799:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629239.585805:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629239.585836:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828740224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629239.585859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629239.585862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629239.585864:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629239.585867:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13574 00000100:00080000:3.0:1778629239.585871:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629239.585874:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629239.585885:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629239.585902:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828740352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629239.585903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828740224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629239.585920:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629239.585922:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629239.585924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629239.585928:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13574 00000100:00080000:3.0:1778629239.585932:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629239.585935:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629239.585954:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629239.585966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828740480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629239.585975:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629239.585981:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1778629239.585982:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629239.585986:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629239.585993:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13574 00000100:00080000:3.0:1778629239.585998:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629239.586008:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629239.586013:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629239.586018:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629239/real 1778629239] req@ffff95434701ea00 x1865013828740224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629239.586029:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828740608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629239.586046:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629239.586049:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629239.586051:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629239.586055:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828740224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629239.586059:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13574 00000100:00080000:3.0:1778629239.586065:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629239.586071:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778629239.586072:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1778629239.586077:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778629239.586079:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629239.586086:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828740224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778629239.586093:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828740736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629239.586118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629239.586149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629239.586163:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629239/real 1778629239] req@ffff95434701c000 x1865013828740352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629239.586175:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828740352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629239.586183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629239.586187:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629239.586190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629239.586195:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629239.586265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629239.586277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629239/real 1778629239] req@ffff95434701c700 x1865013828740480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629239.586291:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828740480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629239.586313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629239.586317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629239.586322:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629239.586329:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629239.586358:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629239.586373:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629239/real 1778629239] req@ffff95434701c380 x1865013828740608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629239.586385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828740608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629239.586395:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629239.586398:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629239.586401:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629239.586406:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629239.586424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629239.586432:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629239/real 1778629239] req@ffff95434701d500 x1865013828740736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629239.586442:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828740736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629239.586450:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629239.586452:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629239.586461:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629239.777772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629239.777838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629240.545769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629240.801754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629240.801806:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629241.569766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629241.825752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629241.825809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629242.593748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629242.849758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629242.849808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629243.617741:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629243.873768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629243.873823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629244.641798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629244.705808:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629244.705818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629244.705822:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629244.705834:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13579 00000100:00080000:1.0:1778629244.705844:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629244.705850:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629244.705884:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828740864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629244.705904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629244.705906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629244.705908:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629244.705910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13579 00000100:00080000:1.0:1778629244.705912:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629244.705915:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629244.705922:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629244.705930:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828740992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629244.705937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629244.705939:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629244.705940:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629244.705945:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13579 00000100:00100000:2.0:1778629244.705946:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828740864:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629244.705962:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629244.705965:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629244.705968:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629244.705978:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828741120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629244.705984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629244.705986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629244.705987:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629244.705990:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13579 00000100:00080000:1.0:1778629244.705992:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629244.705994:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629244.705996:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778629244.706002:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629244.706005:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828741248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629244.706012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629244.706014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629244.706015:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629244.706017:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13579 00000100:00080000:1.0:1778629244.706020:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629244.706022:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629244.706024:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629244.706032:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828741376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629244.706040:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629244/real 1778629244] req@ffff954350194700 x1865013828740864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629244.706074:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828740864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629244.706082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629244.706086:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629244.706088:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828740864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629244.706106:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629244.706121:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629244.706129:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629244/real 1778629244] req@ffff954350197800 x1865013828740992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629244.706137:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828740992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629244.706148:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629244.706150:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629244.706152:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828740992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629244.706156:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629244.706168:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629244.706173:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629244/real 1778629244] req@ffff954350197480 x1865013828741120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629244.706179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828741120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629244.706184:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629244.706186:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629244.706187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629244.706190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629244.706199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629244.706204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629244/real 1778629244] req@ffff954350196d80 x1865013828741248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629244.706210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828741248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629244.706214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629244.706215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629244.706221:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629244.706224:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629244.706235:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629244.706240:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629244/real 1778629244] req@ffff954350194000 x1865013828741376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629244.706247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828741376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629244.706251:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629244.706253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629244.706254:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629244.897746:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629244.897798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629245.665773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629245.921733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629245.921789:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629246.689740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629246.945770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629246.945821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629247.713791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629247.969765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629247.969813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629248.737770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629248.993763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629248.993830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629249.761788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629249.825765:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629249.825776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629249.825781:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629249.825792:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13584 00000100:00080000:1.0:1778629249.825803:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629249.825810:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629249.825854:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828741504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629249.825883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629249.825887:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629249.825889:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629249.825894:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13584 00000100:00080000:1.0:1778629249.825898:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629249.825902:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629249.825912:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629249.825928:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828741632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629249.825938:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828741504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629249.825944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629249.825948:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629249.825963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629249.825968:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13584 00000100:00080000:1.0:1778629249.825973:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629249.825977:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629249.825983:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629249.825996:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828741760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629249.826005:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629249.826009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629249.826014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629249.826016:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629249.826021:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13584 00000100:00080000:1.0:1778629249.826026:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629249.826030:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629249.826035:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629249.826044:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629249/real 1778629249] req@ffff954350197100 x1865013828741504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629249.826048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828741888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629249.826071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629249.826080:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629249.826085:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629249.826092:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13584 00000100:00080000:1.0:1778629249.826095:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629249.826099:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629249.826104:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629249.826106:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828741504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629249.826122:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629249.826127:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629249.826133:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828742016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629249.826134:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828741504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629249.826168:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629249.826194:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629249.826207:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629249/real 1778629249] req@ffff954350194000 x1865013828741632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629249.826220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828741632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629249.826230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629249.826234:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629249.826238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629249.826244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629249.826263:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629249.826273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629249/real 1778629249] req@ffff954350196d80 x1865013828741760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629249.826286:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828741760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629249.826308:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629249.826311:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629249.826315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629249.826322:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629249.826362:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629249.826374:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629249/real 1778629249] req@ffff954350197480 x1865013828741888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629249.826387:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828741888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629249.826396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629249.826400:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629249.826403:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828741888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629249.826409:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629249.826428:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629249.826447:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629249/real 1778629249] req@ffff954350197800 x1865013828742016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629249.826461:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828742016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629249.826470:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629249.826474:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629249.826483:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629250.017786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629250.017855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629250.785766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629251.041752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629251.041968:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629251.809820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629252.065821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629252.065886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629252.833772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629253.089791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629253.089924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629253.857785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629254.113780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629254.113846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629254.881796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629254.945781:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629254.945793:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629254.945798:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629254.945810:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13589 00000100:00080000:1.0:1778629254.945822:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629254.945828:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629254.945873:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828742144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629254.945904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629254.945907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629254.945909:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629254.945912:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13589 00000100:00080000:1.0:1778629254.945914:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629254.945917:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629254.945926:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629254.945941:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828742272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629254.945949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629254.945952:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629254.945953:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629254.945956:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13589 00000100:00080000:1.0:1778629254.945959:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629254.945964:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828742144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629254.945982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629254.945987:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629254.945996:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828742400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629254.946004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629254.946007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629254.946009:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629254.946012:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13589 00000100:00080000:1.0:1778629254.946015:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629254.946017:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629254.946021:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629254.946031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828742528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629254.946036:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629254.946044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629254.946047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629254.946048:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629254.946052:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13589 00000100:00080000:1.0:1778629254.946054:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629254.946057:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629254.946060:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629254.946067:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828742656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629254.946083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629254/real 1778629254] req@ffff954350194700 x1865013828742144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629254.946101:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828742144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629254.946110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629254.946113:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629254.946116:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828742144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629254.946122:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629254.946137:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629254.946145:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629254/real 1778629254] req@ffff954350197b80 x1865013828742272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629254.946153:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828742272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629254.946166:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629254.946168:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629254.946171:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629254.946203:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629254.946226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629254.946237:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629254/real 1778629254] req@ffff954350196a00 x1865013828742400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629254.946259:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828742400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629254.946265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629254.946267:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629254.946269:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629254.946274:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629254.946290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629254.946297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629254/real 1778629254] req@ffff954350194e00 x1865013828742528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629254.946305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828742528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629254.946310:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629254.946313:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629254.946320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629254.946324:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629254.946341:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629254.946352:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629254/real 1778629254] req@ffff954350194a80 x1865013828742656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629254.946364:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828742656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629254.946375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629254.946379:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629254.946382:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629255.137810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629255.138020:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629255.905799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629256.161778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629256.161842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629256.929777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629257.185787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629257.185853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629257.953825:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629258.209814:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629258.209889:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629258.977764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629259.233772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629259.233884:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629260.001784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629260.065801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629260.065826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629260.065830:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629260.065838:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13594 00000100:00080000:1.0:1778629260.065850:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629260.065856:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629260.065888:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828742784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629260.065914:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629260.065918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629260.065920:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629260.065924:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13594 00000100:00080000:1.0:1778629260.065928:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629260.065931:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629260.065940:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629260.065952:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828742912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629260.065963:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629260.065964:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828742784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629260.065967:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629260.065971:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629260.065976:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13594 00000100:00080000:1.0:1778629260.065980:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629260.065983:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629260.065988:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629260.065998:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828743040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629260.066014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629260.066017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629260.066019:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629260.066023:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13594 00000800:00000100:2.0:1778629260.066028:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629260.066030:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629260.066036:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629260.066042:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629260.066054:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828743168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629260.066060:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629260/real 1778629260] req@ffff954350195c00 x1865013828742784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629260.066069:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629260.066073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629260.066074:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629260.066082:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13594 00000100:00080000:1.0:1778629260.066089:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629260.066094:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629260.066099:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629260.066100:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828742784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629260.066110:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778629260.066114:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828743296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629260.066118:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629260.066127:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828742784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629260.066169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629260.066190:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629260.066199:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629260/real 1778629260] req@ffff954350195500 x1865013828742912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629260.066207:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828742912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629260.066213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629260.066215:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629260.066217:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828742912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629260.066220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629260.066230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629260.066235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629260/real 1778629260] req@ffff954350195f80 x1865013828743040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629260.066242:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828743040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629260.066254:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629260.066256:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629260.066258:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629260.066261:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629260.066274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629260.066279:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629260/real 1778629260] req@ffff954350194380 x1865013828743168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629260.066285:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828743168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629260.066290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629260.066292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629260.066293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629260.066296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629260.066304:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629260.066308:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629260/real 1778629260] req@ffff954350195180 x1865013828743296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629260.066314:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828743296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629260.066319:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629260.066320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629260.066327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629260.257760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629260.257830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629261.025766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629261.281778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629261.281822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629262.049765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629262.305758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629262.305802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629263.073783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629263.329774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629263.329850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629264.097777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629264.353747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629264.353809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629265.121834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629265.185759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629265.185766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629265.185769:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629265.185776:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13599 00000100:00080000:1.0:1778629265.185800:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629265.185805:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629265.185831:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828743424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629265.185852:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629265.185855:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629265.185856:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629265.185859:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13599 00000100:00080000:1.0:1778629265.185861:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629265.185863:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629265.185869:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629265.185878:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828743552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629265.185885:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629265.185887:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629265.185889:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629265.185890:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828743424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629265.185893:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13599 00000100:00080000:1.0:1778629265.185897:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629265.185900:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629265.185902:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629265.185909:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828743680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629265.185914:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629265.185920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629265.185921:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629265.185925:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13599 00000100:00080000:1.0:1778629265.185929:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629265.185933:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629265.185937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629265.185947:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828743808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629265.185956:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629265.185960:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629265.185964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629265.185966:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629265.185970:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13599 00000100:00080000:1.0:1778629265.185975:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629265.185979:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629265.185984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629265.185988:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629265/real 1778629265] req@ffff954350195880 x1865013828743424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629265.185995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828743936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629265.186019:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828743424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629265.186030:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629265.186035:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629265.186048:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828743424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629265.186068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629265.186084:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629265.186092:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629265/real 1778629265] req@ffff954350195180 x1865013828743552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629265.186099:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828743552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629265.186109:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629265.186111:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629265.186113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629265.186117:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629265.186128:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629265.186134:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629265/real 1778629265] req@ffff954350194380 x1865013828743680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629265.186141:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828743680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629265.186145:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629265.186147:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629265.186149:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629265.186152:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629265.186162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629265.186169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629265/real 1778629265] req@ffff954350195f80 x1865013828743808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629265.186180:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828743808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629265.186187:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629265.186190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629265.186198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629265.186204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629265.186223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629265.186232:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629265/real 1778629265] req@ffff954350195500 x1865013828743936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629265.186241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828743936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629265.186248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629265.186251:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629265.186253:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828743936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629265.377741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629265.377792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629266.145736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629266.402147:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629266.402893:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629267.169775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629267.425767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629267.425937:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629268.193792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629268.450707:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629268.451358:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629269.217796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629269.474608:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629269.476203:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629270.241793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629270.305816:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629270.305825:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629270.305829:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629270.305839:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13604 00000100:00080000:0.0:1778629270.305857:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629270.305864:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629270.305896:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828744064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629270.305916:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629270.305931:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629270.305933:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629270.305946:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13604 00000100:00080000:0.0:1778629270.305951:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629270.305956:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629270.305958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828744064:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778629270.305964:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629270.305976:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828744192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629270.305984:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629270.305986:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629270.305987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629270.305989:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13604 00000100:00080000:0.0:1778629270.305992:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629270.305993:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629270.305996:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629270.306002:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828744320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629270.306007:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629270.306009:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629270.306010:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629270.306012:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13604 00000100:00080000:0.0:1778629270.306015:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778629270.306019:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629270.306021:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629270.306029:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629270.306037:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828744448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629270.306043:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629270.306045:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629270.306046:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629270.306050:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13604 00000100:00100000:2.0:1778629270.306051:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629270/real 1778629270] req@ffff954342f49880 x1865013828744064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629270.306053:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629270.306058:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629270.306063:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629270.306068:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828744576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629270.306079:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828744064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629270.306097:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629270.306100:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629270.306103:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828744064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629270.306120:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629270.306134:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629270.306141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629270/real 1778629270] req@ffff954342f4bb80 x1865013828744192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629270.306148:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828744192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629270.306152:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629270.306154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629270.306156:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629270.306159:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629270.306167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629270.306171:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629270/real 1778629270] req@ffff954342f48e00 x1865013828744320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629270.306177:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828744320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629270.306182:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629270.306183:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629270.306184:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629270.306192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629270.306203:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629270.306208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629270/real 1778629270] req@ffff954342f4aa00 x1865013828744448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629270.306214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828744448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629270.306219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629270.306220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629270.306222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629270.306225:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629270.306232:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629270.306237:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629270/real 1778629270] req@ffff954342f49180 x1865013828744576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629270.306243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828744576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629270.306247:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629270.306249:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629270.306250:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629270.497845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629270.498837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629271.265837:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629271.521946:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629271.522156:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629272.289767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629272.545750:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629272.545799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629273.313767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629273.569759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629273.569821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629274.337775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629274.593772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629274.593822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629275.361776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629275.425766:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629275.425774:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629275.425777:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629275.425785:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13609 00000100:00080000:3.0:1778629275.425794:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629275.425799:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629275.425830:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828744704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629275.425865:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629275.425884:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629275.425885:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629275.425889:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13609 00000100:00080000:3.0:1778629275.425893:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629275.425896:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629275.425904:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828744704:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778629275.425905:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629275.425917:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828744832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629275.425925:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629275.425927:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629275.425928:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629275.425930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13609 00000100:00080000:3.0:1778629275.425932:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629275.425934:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629275.425937:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629275.425944:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828744960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629275.425949:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629275.425951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629275.425952:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629275.425954:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13609 00000100:00080000:3.0:1778629275.425956:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629275.425958:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629275.425964:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778629275.425973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629275.425985:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828745088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629275.425994:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629275/real 1778629275] req@ffff95434701f100 x1865013828744704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629275.425997:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629275.426006:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629275.426009:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629275.426013:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13609 00000100:00100000:2.0:1778629275.426016:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828744704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629275.426020:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629275.426025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778629275.426027:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778629275.426030:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629275.426035:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828744704:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778629275.426036:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629275.426050:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828745216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629275.426053:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629275.426073:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629275.426082:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629275/real 1778629275] req@ffff95434701d880 x1865013828744832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629275.426089:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828744832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629275.426095:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629275.426097:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629275.426098:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629275.426114:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629275.426129:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629275.426135:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629275/real 1778629275] req@ffff95434701f800 x1865013828744960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629275.426142:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828744960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629275.426146:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629275.426148:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629275.426150:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828744960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629275.426153:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629275.426162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629275.426167:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629275/real 1778629275] req@ffff95434701d500 x1865013828745088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629275.426173:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828745088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629275.426177:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629275.426179:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629275.426180:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629275.426190:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629275.426198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629275.426202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629275/real 1778629275] req@ffff95434701c380 x1865013828745216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629275.426214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828745216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629275.426219:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629275.426220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629275.426222:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629275.617780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629275.617851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629276.385769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629276.641773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629276.641833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629277.409774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629277.665781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629277.665841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629278.433739:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629278.689782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629278.689836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629279.457791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629279.713781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629279.713831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629280.481787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629280.545789:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629280.545798:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629280.545803:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629280.545811:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13615 00000100:00080000:3.0:1778629280.545822:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629280.545828:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629280.545858:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828745344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629280.545883:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629280.545886:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629280.545888:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629280.545892:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13615 00000100:00080000:3.0:1778629280.545908:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629280.545912:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629280.545921:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629280.545927:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828745344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778629280.545934:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828745472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629280.545945:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629280.545948:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629280.545950:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629280.545954:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13615 00000100:00080000:3.0:1778629280.545957:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629280.545960:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629280.545964:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629280.545974:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828745600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629280.545981:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629280.545987:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629280.545991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629280.545993:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629280.545997:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13615 00000100:00080000:3.0:1778629280.546000:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629280.546004:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629280.546008:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629280.546012:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629280/real 1778629280] req@ffff95434701c700 x1865013828745344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629280.546020:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828745728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629280.546029:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828745344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629280.546056:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778629280.546057:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629280.546060:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778629280.546063:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778629280.546065:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828745344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629280.546066:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629280.546070:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13615 00000100:00080000:3.0:1778629280.546075:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629280.546079:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629280.546084:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629280.546094:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828745856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629280.546096:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629280.546124:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629280.546139:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629280/real 1778629280] req@ffff95434701c000 x1865013828745472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629280.546151:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828745472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629280.546159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629280.546162:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629280.546165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629280.546170:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629280.546189:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629280.546197:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629280/real 1778629280] req@ffff95434701ea00 x1865013828745600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629280.546204:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828745600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629280.546209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629280.546211:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629280.546213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629280.546216:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629280.546227:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629280.546231:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629280/real 1778629280] req@ffff95434701f480 x1865013828745728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629280.546243:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828745728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629280.546248:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629280.546250:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629280.546251:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629280.546261:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629280.546272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629280.546277:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629280/real 1778629280] req@ffff95434701df80 x1865013828745856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629280.546283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828745856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629280.546288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629280.546289:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629280.546290:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828745856:192.168.204.102@tcp:8:kworker.0 00010000:00010000:1.0:1778629280.738031:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629280.738356:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629281.505803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629281.761790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629281.761850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629282.529742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629282.785799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629282.785870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629283.553811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629283.809797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629283.809858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629284.577786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629284.833755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629284.833816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629285.601765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629285.665813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629285.665820:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629285.665824:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629285.665831:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13620 00000100:00080000:1.0:1778629285.665840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629285.665846:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629285.665892:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828745984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629285.665915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629285.665918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629285.665919:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629285.665922:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13620 00000100:00080000:1.0:1778629285.665924:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629285.665927:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629285.665933:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629285.665942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828746112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629285.665949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629285.665963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1778629285.665963:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828745984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629285.665966:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629285.665972:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13620 00000100:00080000:1.0:1778629285.665975:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629285.665977:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629285.665980:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629285.665988:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828746240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629285.665995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629285.665997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629285.665998:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629285.666001:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13620 00000100:00080000:1.0:1778629285.666004:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629285.666006:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629285.666008:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629285.666015:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828746368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629285.666021:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629285.666027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1778629285.666027:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629285.666031:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629285.666035:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13620 00000100:00080000:1.0:1778629285.666039:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629285.666043:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629285.666053:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629285.666063:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629285/real 1778629285] req@ffff954350195c00 x1865013828745984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629285.666067:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828746496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629285.666094:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828745984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629285.666105:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629285.666109:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629285.666113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828745984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629285.666121:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629285.666140:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629285.666151:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629285/real 1778629285] req@ffff954350194a80 x1865013828746112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629285.666162:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828746112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629285.666171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629285.666174:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629285.666177:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629285.666198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629285.666215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629285.666225:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629285/real 1778629285] req@ffff954350194e00 x1865013828746240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629285.666256:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828746240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629285.666264:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629285.666267:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629285.666271:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629285.666276:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629285.666294:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629285.666304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629285/real 1778629285] req@ffff954350196a00 x1865013828746368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629285.666315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828746368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629285.666322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629285.666325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629285.666328:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629285.666334:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629285.666350:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629285.666359:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629285/real 1778629285] req@ffff954350197b80 x1865013828746496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629285.666370:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828746496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629285.666389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629285.666392:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629285.666395:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629285.857818:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629285.857910:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629286.625778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629286.881741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629286.881810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629287.649765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629287.905771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629287.905825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629288.673776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629288.929780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629288.929908:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629289.697777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629289.953733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629289.953795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629290.721770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629290.785757:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629290.785764:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629290.785767:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629290.785774:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13625 00000100:00080000:0.0:1778629290.785783:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629290.785788:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629290.785815:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828746624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629290.785832:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629290.785834:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629290.785836:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629290.785838:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13625 00000100:00080000:0.0:1778629290.785852:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629290.785856:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629290.785863:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629290.785872:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828746752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629290.785876:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828746624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629290.785880:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629290.785885:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629290.785886:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629290.785889:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13625 00000100:00080000:0.0:1778629290.785891:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629290.785892:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629290.785895:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629290.785902:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828746880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629290.785907:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629290.785909:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629290.785910:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629290.785913:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13625 00000100:00080000:0.0:1778629290.785914:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629290.785916:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629290.785919:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629290.785924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828747008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629290.785937:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629290.785942:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1778629290.785942:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629290.785945:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629290.785949:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13625 00000100:00080000:0.0:1778629290.785952:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629290.785955:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629290.785958:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629290.785967:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828747136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629290.785973:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629290/real 1778629290] req@ffff954342f48000 x1865013828746624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629290.785988:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828746624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629290.785995:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629290.785998:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629290.786000:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828746624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629290.786017:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629290.786038:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629290.786057:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629290/real 1778629290] req@ffff954342f48380 x1865013828746752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629290.786068:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828746752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629290.786076:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629290.786080:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629290.786083:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629290.786089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629290.786107:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629290.786115:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629290/real 1778629290] req@ffff954342f49180 x1865013828746880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629290.786126:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828746880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629290.786133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629290.786136:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629290.786139:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828746880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629290.786145:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629290.786164:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629290.786172:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629290/real 1778629290] req@ffff954342f4aa00 x1865013828747008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629290.786179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828747008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629290.786188:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629290.786190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629290.786192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629290.786195:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629290.786207:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629290.786212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629290/real 1778629290] req@ffff954342f48e00 x1865013828747136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629290.786219:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828747136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629290.786223:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629290.786225:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629290.786226:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629290.977770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629290.977829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629291.745754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629292.001740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629292.001781:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629292.769771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629293.025755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629293.025819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629293.793759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629294.049757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629294.049808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629294.817775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629295.073779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629295.073828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629295.841767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629295.905790:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629295.905800:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629295.905804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629295.905811:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13630 00000100:00080000:3.0:1778629295.905820:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629295.905825:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629295.905854:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828747264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629295.905877:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629295.905879:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629295.905881:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629295.905883:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13630 00000100:00080000:3.0:1778629295.905886:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629295.905888:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629295.905906:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629295.905915:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828747392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629295.905923:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629295.905925:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629295.905926:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629295.905930:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13630 00000100:00100000:2.0:1778629295.905932:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828747264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629295.905933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629295.905938:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629295.905942:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629295.905952:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828747520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629295.905971:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629295.905974:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629295.905977:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629295.905980:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13630 00000100:00080000:3.0:1778629295.905993:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629295.905998:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629295.906002:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778629295.906007:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778629295.906014:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828747648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629295.906021:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629295.906024:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629295.906029:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629295.906032:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13630 00000100:00080000:3.0:1778629295.906035:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629295.906039:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629295.906041:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629295/real 1778629295] req@ffff95434701e680 x1865013828747264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778629295.906043:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629295.906054:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828747776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629295.906067:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828747264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629295.906077:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629295.906081:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629295.906084:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828747264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629295.906103:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629295.906123:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629295.906134:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629295/real 1778629295] req@ffff95434701ca80 x1865013828747392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629295.906158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828747392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629295.906166:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629295.906169:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629295.906172:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629295.906178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629295.906197:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629295.906208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629295/real 1778629295] req@ffff95434701d180 x1865013828747520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629295.906232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828747520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629295.906240:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629295.906243:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629295.906246:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629295.906252:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629295.906272:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629295.906282:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629295/real 1778629295] req@ffff95434701df80 x1865013828747648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629295.906293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828747648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629295.906300:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629295.906303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629295.906306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629295.906319:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629295.906337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629295.906347:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629295/real 1778629295] req@ffff95434701f480 x1865013828747776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629295.906357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828747776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629295.906375:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629295.906378:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629295.906381:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828747776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629296.097754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629296.097810:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629296.865809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629297.121791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629297.121850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629297.889748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629298.145767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629298.145819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629298.913785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629299.169763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629299.169789:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629299.937759:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629300.193774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629300.193823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629300.961775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629301.025770:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629301.025780:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629301.025784:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629301.025794:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13635 00000100:00080000:3.0:1778629301.025805:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629301.025812:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629301.025849:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828747904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629301.025881:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629301.025884:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629301.025885:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629301.025888:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13635 00000100:00080000:3.0:1778629301.025891:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629301.025893:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629301.025900:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629301.025910:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828748032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629301.025932:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828747904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629301.025937:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629301.025940:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629301.025941:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629301.025944:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13635 00000100:00080000:3.0:1778629301.025948:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629301.025950:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629301.025953:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629301.025963:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828748160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629301.025968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629301.025970:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629301.025972:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629301.025974:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13635 00000100:00080000:3.0:1778629301.025977:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629301.025979:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629301.025981:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629301.025988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828748288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629301.025994:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629301.025997:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629301.025998:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629301.025999:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13635 00000100:00080000:3.0:1778629301.026006:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629301.026010:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629301.026015:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629301.026026:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828748416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629301.026029:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629301.026056:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629301.026068:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629301.026086:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629301.217757:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629301.217807:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629301.985771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629302.241753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629302.241791:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629303.009763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629303.265767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629303.265826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629304.033802:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778629304.097972:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778629304.098011:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778629304.098027:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629304.098065:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629304.098072:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629304.098079:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629304.098086:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778629304.098156:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629301/real 1778629304] req@ffff95434701d500 x1865013828748416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629304.098181:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828748416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629304.098192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629304.098197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629304.098201:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629304.098208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629301/real 1778629304] req@ffff95434701c000 x1865013828748032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629304.098219:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828748032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629304.098241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629304.098245:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629304.098249:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629304.098255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629301/real 1778629304] req@ffff95434701c700 x1865013828748160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629304.098267:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828748160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629304.098288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629304.098292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629304.098295:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629304.098302:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629301/real 1778629304] req@ffff95434701c380 x1865013828748288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629304.098313:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828748288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629304.098320:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629304.098323:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629304.098326:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629304.098332:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629301/real 1778629304] req@ffff95434701ea00 x1865013828747904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629304.098342:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828747904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629304.098355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629304.098358:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629304.098361:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828747904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629304.289786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629304.289831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629305.057783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629305.313758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629305.313859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629306.081822:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629306.145835:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629306.145848:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629306.145851:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629306.145859:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13640 00000100:00080000:1.0:1778629306.145875:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629306.145881:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629306.145932:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828748544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629306.145976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629306.145979:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629306.145993:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629306.145999:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13640 00000100:00080000:1.0:1778629306.146003:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629306.146007:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629306.146018:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629306.146034:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828748672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629306.146035:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828748544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629306.146051:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629306.146054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629306.146098:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629306.146103:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13640 00000100:00080000:1.0:1778629306.146108:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778629306.146118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629306.146127:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629306.146134:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629306.146155:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629306/real 1778629306] req@ffff954350194700 x1865013828748544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629306.146159:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828748800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629306.146188:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828748544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629306.146190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629306.146196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629306.146199:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629306.146213:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629306.146218:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13640 00000100:00080000:2.0:1778629306.146221:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629306.146225:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629306.146228:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828748544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629306.146253:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629306.146260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629306.146274:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828748928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629306.146277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629306.146305:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629306.146330:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629306/real 1778629306] req@ffff954350197800 x1865013828748672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629306.146333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629306.146356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1778629306.146359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828748672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629306.146360:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629306.146370:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629306.146373:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629306.146377:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629306.146382:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748800:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778629306.146388:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13640 00000100:00080000:1.0:1778629306.146393:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629306.146397:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629306.146407:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778629306.146417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629306.146419:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629306/real 1778629306] req@ffff954350197480 x1865013828748800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629306.146434:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828748800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629306.146442:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629306.146446:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629306.146450:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629306.146467:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629306.146470:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828749056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629306.146489:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629306.146501:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629306/real 1778629306] req@ffff954350196d80 x1865013828748928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629306.146513:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828748928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629306.146531:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629306.146535:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629306.146538:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828748928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629306.146551:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629306.146571:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629306.146581:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629306/real 1778629306] req@ffff954350194000 x1865013828749056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629306.146592:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828749056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629306.146599:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629306.146602:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629306.146605:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629306.337733:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629306.337776:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629307.105760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629307.361751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629307.361800:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629308.129812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629308.385773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629308.385820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629309.153820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629309.409749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629309.409816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629310.177795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629310.433753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629310.433805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629311.201772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629311.265783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629311.265793:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629311.265797:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629311.265807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13645 00000100:00080000:1.0:1778629311.265818:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629311.265824:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629311.265859:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828749184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629311.265886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629311.265889:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629311.265891:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629311.265895:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13645 00000100:00080000:1.0:1778629311.265899:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629311.265902:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629311.265911:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629311.265924:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828749312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629311.265930:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828749184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629311.265941:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629311.265944:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629311.265946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629311.265952:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13645 00000100:00080000:1.0:1778629311.265956:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629311.265968:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629311.265973:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629311.265985:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828749440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629311.265993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629311.265997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629311.265999:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778629311.265999:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629311.266006:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13645 00000100:00080000:1.0:1778629311.266012:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629311.266017:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629311.266022:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629311.266031:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629311/real 1778629311] req@ffff954350197100 x1865013828749184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629311.266034:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828749568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629311.266057:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629311.266060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629311.266062:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629311.266067:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13645 00000100:00100000:2.0:1778629311.266067:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828749184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629311.266073:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629311.266080:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629311.266086:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778629311.266086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629311.266095:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629311.266099:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828749696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629311.266101:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828749184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629311.266143:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749312:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629311.266166:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629311.266175:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629311/real 1778629311] req@ffff954350194000 x1865013828749312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629311.266185:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828749312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629311.266193:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629311.266196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629311.266199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629311.266204:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629311.266222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629311.266232:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629311/real 1778629311] req@ffff954350196d80 x1865013828749440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629311.266256:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828749440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629311.266261:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629311.266263:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629311.266265:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629311.266269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629311.266284:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629311.266290:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629311/real 1778629311] req@ffff954350197480 x1865013828749568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629311.266297:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828749568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629311.266301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629311.266303:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629311.266305:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629311.266307:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629311.266315:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629311.266320:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629311/real 1778629311] req@ffff954350197800 x1865013828749696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629311.266329:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828749696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629311.266347:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629311.266351:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629311.266354:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629311.457728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629311.457767:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629312.225785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629312.481739:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629312.481825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629313.249790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629313.505741:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629313.505803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629314.273791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629314.529765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629314.529823:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629315.297770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629315.553738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629315.553798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629316.321785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629316.385787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629316.385809:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629316.385812:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629316.385820:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13650 00000100:00080000:1.0:1778629316.385828:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629316.385834:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629316.385860:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828749824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629316.385882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629316.385884:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629316.385885:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629316.385888:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13650 00000100:00080000:1.0:1778629316.385890:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629316.385891:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629316.385904:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629316.385916:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828749952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629316.385927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629316.385930:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629316.385933:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629316.385936:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828749824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629316.385938:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13650 00000100:00080000:1.0:1778629316.385944:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629316.385948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629316.385952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629316.385961:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828750080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629316.385972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629316.385974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629316.385975:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629316.385978:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13650 00000100:00080000:1.0:1778629316.385980:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629316.385983:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629316.385986:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629316.385992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828750208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629316.385997:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629316.386001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1778629316.386002:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629316.386003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629316.386007:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13650 00000100:00080000:1.0:1778629316.386010:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629316.386012:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629316.386014:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629316.386020:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828750336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629316.386058:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629316/real 1778629316] req@ffff954350194700 x1865013828749824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629316.386078:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828749824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629316.386088:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629316.386093:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629316.386096:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828749824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629316.386119:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629316.386149:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629316.386172:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629316/real 1778629316] req@ffff954350197b80 x1865013828749952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629316.386186:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828749952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629316.386194:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629316.386197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629316.386200:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828749952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629316.386206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629316.386226:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629316.386236:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629316/real 1778629316] req@ffff954350196a00 x1865013828750080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629316.386247:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828750080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629316.386255:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629316.386258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629316.386261:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629316.386267:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629316.386285:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629316.386294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629316/real 1778629316] req@ffff954350194e00 x1865013828750208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629316.386306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828750208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629316.386321:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629316.386324:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629316.386327:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629316.386332:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629316.386352:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629316.386372:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629316/real 1778629316] req@ffff954350194a80 x1865013828750336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629316.386385:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828750336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629316.386392:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629316.386395:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629316.386398:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629316.577776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629316.577829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629317.345760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629317.601801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629317.601868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629318.369779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629318.625768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629318.625815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629319.393764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629319.649764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629319.649826:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629320.417736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629320.673766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629320.673827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629321.441746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629321.505769:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629321.505779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629321.505800:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629321.505809:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13656 00000100:00080000:2.0:1778629321.505820:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629321.505826:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629321.505856:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828750464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629321.505886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629321.505890:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629321.505892:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629321.505896:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13656 00000100:00080000:2.0:1778629321.505899:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629321.505902:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629321.505911:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629321.505922:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828750592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629321.505946:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828750464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629321.505946:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629321.505952:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629321.505956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629321.505962:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13656 00000100:00080000:2.0:1778629321.505965:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629321.505969:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629321.505974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629321.505985:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828750720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629321.506007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629321.506010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629321.506012:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778629321.506017:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629321.506019:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13656 00000100:00080000:2.0:1778629321.506025:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629321.506030:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629321.506035:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629321.506047:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828750848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629321.506056:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629321/real 1778629321] req@ffff954358788e00 x1865013828750464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629321.506069:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629321.506073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629321.506075:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629321.506081:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13656 00000100:00080000:2.0:1778629321.506087:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629321.506093:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629321.506097:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629321.506104:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828750464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629321.506114:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1778629321.506116:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828750976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629321.506122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629321.506130:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828750464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629321.506170:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629321.506193:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629321.506205:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629321/real 1778629321] req@ffff954358789c00 x1865013828750592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629321.506218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828750592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629321.506225:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629321.506229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629321.506232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629321.506238:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629321.506257:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629321.506267:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629321/real 1778629321] req@ffff954358788380 x1865013828750720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629321.506278:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828750720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629321.506297:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629321.506300:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629321.506303:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629321.506309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629321.506328:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629321.506339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629321/real 1778629321] req@ffff95435878aa00 x1865013828750848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629321.506350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828750848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629321.506358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629321.506361:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629321.506364:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629321.506370:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629321.506388:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629321.506398:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629321/real 1778629321] req@ffff95435878b100 x1865013828750976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629321.506408:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828750976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629321.506415:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629321.506418:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629321.506427:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828750976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629321.697800:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629321.697861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629322.465771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629322.721760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629322.721822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629323.489832:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629323.745760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629323.745831:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629324.513833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629324.769778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629324.769833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629325.537793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629325.793751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629325.793798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629326.561756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629326.625799:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629326.625806:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629326.625810:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629326.625816:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13661 00000100:00080000:0.0:1778629326.625836:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629326.625842:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629326.625871:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828751104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629326.625891:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629326.625895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629326.625896:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629326.625900:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13661 00000100:00080000:0.0:1778629326.625919:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629326.625923:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629326.625931:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629326.625938:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828751104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629326.625943:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828751232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629326.625951:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629326.625953:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629326.625955:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629326.625958:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13661 00000100:00080000:0.0:1778629326.625960:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629326.625962:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629326.625966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629326.625973:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828751360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629326.625980:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629326.625982:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629326.625984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629326.625987:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13661 00000100:00080000:0.0:1778629326.625989:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629326.625992:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629326.625993:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778629326.625995:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629326.626002:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828751488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629326.626022:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629326.626023:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629326/real 1778629326] req@ffff954342f4bb80 x1865013828751104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629326.626025:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629326.626026:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629326.626030:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13661 00000100:00080000:0.0:1778629326.626033:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629326.626036:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629326.626038:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828751104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778629326.626040:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:0.0:1778629326.626049:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828751616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629326.626057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629326.626060:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629326.626063:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828751104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629326.626084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629326.626103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629326.626112:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629326/real 1778629326] req@ffff954342f49880 x1865013828751232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629326.626120:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828751232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629326.626125:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629326.626127:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629326.626129:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629326.626132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629326.626142:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629326.626153:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629326/real 1778629326] req@ffff954342f4a300 x1865013828751360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629326.626159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828751360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629326.626164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629326.626165:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629326.626167:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629326.626170:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629326.626181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629326.626185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629326/real 1778629326] req@ffff954342f4b480 x1865013828751488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629326.626191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828751488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629326.626195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629326.626196:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629326.626198:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629326.626200:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629326.626208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629326.626212:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629326/real 1778629326] req@ffff954342f49c00 x1865013828751616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629326.626218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828751616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629326.626226:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629326.626228:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629326.626229:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629326.817764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629326.817816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629327.585791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629327.841762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629327.841853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629328.609758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629328.865751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629328.865801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629329.633793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629329.889784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629329.889861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629330.657727:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629330.913748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629330.913801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629331.681775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629331.745762:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629331.745772:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629331.745776:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629331.745785:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13666 00000100:00080000:1.0:1778629331.745796:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629331.745802:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629331.745834:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828751744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629331.745860:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629331.745864:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629331.745865:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629331.745869:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13666 00000100:00080000:1.0:1778629331.745873:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629331.745877:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629331.745886:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629331.745896:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828751744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629331.745900:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828751872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629331.745912:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629331.745915:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629331.745931:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629331.745935:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13666 00000100:00080000:1.0:1778629331.745940:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629331.745944:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629331.745947:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778629331.745951:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629331.745972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828752000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629331.745976:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629331/real 1778629331] req@ffff954350195c00 x1865013828751744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629331.745999:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629331.746002:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1778629331.746004:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828751744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629331.746005:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629331.746014:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13666 00000100:00080000:2.0:1778629331.746016:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629331.746019:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629331.746021:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629331.746027:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629331.746033:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629331.746034:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828751744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629331.746047:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828752128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629331.746054:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751872:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778629331.746059:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629331.746063:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629331.746065:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629331.746071:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13666 00000800:00000100:2.0:1778629331.746071:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629331.746077:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629331.746082:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629331.746083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629331/real 1778629331] req@ffff954350195500 x1865013828751872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778629331.746088:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629331.746099:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828751872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629331.746106:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778629331.746108:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828752256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629331.746110:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629331.746115:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828751872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629331.746125:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629331.746141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629331.746150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629331/real 1778629331] req@ffff954350195f80 x1865013828752000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629331.746161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828752000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629331.746168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629331.746171:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629331.746174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629331.746187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629331.746205:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629331.746224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629331/real 1778629331] req@ffff954350194380 x1865013828752128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629331.746231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828752128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629331.746244:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629331.746246:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629331.746248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629331.746251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629331.746264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629331.746269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629331/real 1778629331] req@ffff954350195180 x1865013828752256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629331.746276:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828752256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629331.746281:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629331.746282:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629331.746284:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629331.937780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629331.937853:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629332.705843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629332.961812:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629332.961858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629333.729792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629333.985898:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629333.985991:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629334.753788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629335.010124:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1778629335.010321:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629335.777805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629336.034171:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629336.034190:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629336.801736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629336.865836:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629336.865842:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629336.865846:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629336.865853:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13671 00000100:00080000:2.0:1778629336.865881:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629336.865886:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629336.865912:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828752384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629336.865933:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629336.865935:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629336.865937:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629336.865939:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13671 00000100:00080000:2.0:1778629336.865941:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629336.865943:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629336.865950:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629336.865962:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828752512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629336.865968:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828752384:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629336.865975:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629336.865979:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629336.865980:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629336.865985:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13671 00000100:00080000:2.0:1778629336.865989:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629336.865993:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629336.865998:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629336.866009:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828752640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629336.866015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778629336.866021:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629336.866023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629336.866024:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629336.866028:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13671 00000100:00080000:2.0:1778629336.866032:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629336.866035:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629336.866039:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629336.866048:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828752768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629336.866050:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629336/real 1778629336] req@ffff95435878b480 x1865013828752384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629336.866062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629336.866066:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629336.866067:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629336.866077:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13671 00000100:00100000:1.0:1778629336.866078:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828752384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629336.866079:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629336.866084:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629336.866090:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778629336.866093:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629336.866098:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629336.866100:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828752896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629336.866104:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828752384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629336.866130:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629336.866143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629336.866150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629336/real 1778629336] req@ffff95435878bb80 x1865013828752512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629336.866157:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828752512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629336.866161:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629336.866163:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629336.866165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752512:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629336.866176:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752640:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629336.866187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629336.866192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629336/real 1778629336] req@ffff95435878b100 x1865013828752640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629336.866198:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828752640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629336.866203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629336.866204:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629336.866206:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629336.866209:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629336.866217:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629336.866221:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629336/real 1778629336] req@ffff95435878aa00 x1865013828752768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629336.866232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828752768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629336.866237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629336.866238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629336.866240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629336.866242:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629336.866251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629336.866255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629336/real 1778629336] req@ffff954358788380 x1865013828752896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629336.866266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828752896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629336.866270:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629336.866272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629336.866274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828752896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629337.057820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629337.057862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629337.825762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1778629338.082338:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629338.082789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629338.849751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629339.105763:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629339.105797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629339.873764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629340.129760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629340.129795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629340.897783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629341.153742:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629341.153772:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629341.921823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629341.985764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629341.985779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629341.985803:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629341.985815:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13676 00000100:00080000:2.0:1778629341.985828:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629341.985835:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629341.985873:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828753024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629341.985901:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629341.985904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629341.985906:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629341.985909:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13676 00000100:00080000:2.0:1778629341.985911:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629341.985914:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629341.985922:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629341.985932:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828753152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629341.985940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629341.985943:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629341.985944:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629341.985946:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13676 00000100:00080000:2.0:1778629341.985948:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629341.985951:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629341.985955:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629341.985963:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828753280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629341.985978:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828753024:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629341.985985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629341.985989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629341.985990:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629341.985995:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13676 00000100:00080000:2.0:1778629341.986000:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629341.986006:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629341.986012:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629341.986028:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828753408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629341.986038:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629341.986043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629341.986045:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629341.986050:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13676 00000100:00080000:2.0:1778629341.986054:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778629341.986055:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629341.986064:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629341.986071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629341.986084:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828753536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629341.986104:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629341/real 1778629341] req@ffff954358789c00 x1865013828753024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629341.986121:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828753024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629341.986130:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629341.986133:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629341.986136:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828753024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629341.986147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629341.986178:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629341.986187:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629341/real 1778629341] req@ffff954358788e00 x1865013828753152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629341.986195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828753152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629341.986201:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629341.986203:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629341.986205:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753152:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629341.986227:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629341.986246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629341.986257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629341/real 1778629341] req@ffff954358788700 x1865013828753280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629341.986270:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828753280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629341.986278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629341.986281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629341.986283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629341.986288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629341.986303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629341.986311:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629341/real 1778629341] req@ffff954358789880 x1865013828753408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629341.986328:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828753408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629341.986334:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629341.986336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629341.986338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753408:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629341.986342:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629341.986354:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629341.986360:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629341/real 1778629341] req@ffff954358789500 x1865013828753536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629341.986367:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828753536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629341.986373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629341.986374:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629341.986376:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629342.177789:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629342.177876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629342.945838:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629343.201783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629343.201851:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629343.969793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629344.225762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629344.225813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629344.993845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629345.249759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629345.249836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629346.017749:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629346.273771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629346.273832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629347.041773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629347.105797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629347.105807:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629347.105812:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629347.105839:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13681 00000100:00080000:1.0:1778629347.105851:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629347.105857:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629347.105889:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828753664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629347.105919:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629347.105923:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629347.105925:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629347.105930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13681 00000100:00080000:1.0:1778629347.105935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629347.105939:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629347.105949:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629347.105949:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828753664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629347.105971:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828753792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629347.105984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629347.105988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629347.105990:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629347.105993:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13681 00000100:00080000:1.0:1778629347.106000:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629347.106004:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629347.106010:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778629347.106013:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629347.106029:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828753920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629347.106053:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629347/real 1778629347] req@ffff954350195880 x1865013828753664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629347.106060:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629347.106069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629347.106073:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629347.106080:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13681 00000100:00080000:1.0:1778629347.106086:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629347.106087:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828753664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629347.106094:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629347.106105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:2.0:1778629347.106108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629347.106114:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629347.106120:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828753664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629347.106139:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828754048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629347.106150:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629347.106154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629347.106156:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629347.106160:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13681 00000100:00100000:2.0:1778629347.106163:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753792:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778629347.106167:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629347.106174:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629347.106180:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778629347.106188:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629347.106195:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828754176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629347.106202:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629347/real 1778629347] req@ffff954350195180 x1865013828753792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629347.106222:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828753792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629347.106229:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629347.106232:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629347.106235:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629347.106240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629347.106251:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629347.106256:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629347/real 1778629347] req@ffff954350194380 x1865013828753920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629347.106274:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828753920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629347.106280:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629347.106282:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629347.106284:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828753920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629347.106287:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629347.106303:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629347.106313:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629347/real 1778629347] req@ffff954350195f80 x1865013828754048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629347.106325:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828754048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629347.106333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629347.106337:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629347.106340:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629347.106358:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629347.106385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629347.106392:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629347/real 1778629347] req@ffff954350195500 x1865013828754176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629347.106400:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828754176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629347.106414:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629347.106416:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629347.106418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629347.297783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629347.297848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629348.065744:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629348.321777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629348.321835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629349.089806:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629349.345784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629349.345854:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629350.113814:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629350.369773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629350.369819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629351.137824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629351.393820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629351.393913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629352.161784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629352.225777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629352.225785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629352.225789:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629352.225797:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13686 00000100:00080000:1.0:1778629352.225806:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629352.225830:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629352.225869:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828754304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629352.225891:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629352.225894:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629352.225895:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629352.225899:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13686 00000100:00080000:1.0:1778629352.225907:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629352.225911:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629352.225921:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629352.225936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828754432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629352.225949:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629352.225950:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828754304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629352.225956:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629352.225961:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629352.225965:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13686 00000100:00080000:1.0:1778629352.225968:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629352.225978:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629352.225984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629352.226012:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828754560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629352.226024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1778629352.226025:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629352.226031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629352.226033:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629352.226039:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13686 00000100:00080000:1.0:1778629352.226043:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629352.226047:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629352.226051:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629352.226063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828754688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629352.226070:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629352/real 1778629352] req@ffff954350195c00 x1865013828754304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629352.226071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629352.226078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629352.226081:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629352.226088:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13686 00000100:00080000:1.0:1778629352.226093:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629352.226096:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629352.226099:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629352.226107:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828754304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629352.226108:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828754816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629352.226139:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629352.226145:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629352.226151:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828754304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629352.226176:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629352.226209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629352.226224:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629352/real 1778629352] req@ffff954350194a80 x1865013828754432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629352.226237:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828754432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629352.226253:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629352.226258:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629352.226262:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629352.226268:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629352.226292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629352.226304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629352/real 1778629352] req@ffff954350194e00 x1865013828754560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629352.226315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828754560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629352.226335:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629352.226339:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629352.226342:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629352.226349:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629352.226370:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629352.226381:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629352/real 1778629352] req@ffff954350196a00 x1865013828754688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629352.226395:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828754688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629352.226403:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629352.226406:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629352.226416:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629352.226423:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629352.226447:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629352.226458:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629352/real 1778629352] req@ffff954350197b80 x1865013828754816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629352.226472:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828754816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629352.226481:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629352.226485:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629352.226488:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828754816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629352.417760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629352.417811:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629353.185820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629353.441748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629353.441802:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629354.209768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629354.465753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629354.465815:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629355.233787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629355.489765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629355.489833:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629356.257772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629356.513717:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629356.513808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629357.281763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629357.345792:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629357.345800:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629357.345804:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629357.345813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13691 00000100:00080000:0.0:1778629357.345824:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629357.345830:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629357.345863:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828754944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629357.345886:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629357.345890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629357.345891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629357.345894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13691 00000100:00080000:0.0:1778629357.345910:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629357.345915:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629357.345923:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629357.345928:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828754944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629357.345937:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828755072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629357.345948:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629357.345951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629357.345953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629357.345956:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13691 00000100:00080000:0.0:1778629357.345959:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629357.345962:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629357.345966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629357.345974:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828755200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629357.345983:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629357.345985:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629357.345987:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778629357.345993:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629357.345995:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13691 00000100:00080000:0.0:1778629357.345999:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629357.346002:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629357.346006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629357.346016:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828755328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629357.346028:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629357/real 1778629357] req@ffff954342f49500 x1865013828754944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629357.346043:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629357.346047:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629357.346049:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629357.346053:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13691 00000100:00080000:0.0:1778629357.346057:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629357.346060:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828754944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629357.346063:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629357.346069:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778629357.346075:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778629357.346080:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828755456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629357.346081:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629357.346094:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828754944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629357.346118:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629357.346137:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629357.346146:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629357/real 1778629357] req@ffff954342f4ad80 x1865013828755072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629357.346154:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828755072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629357.346159:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629357.346161:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629357.346163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629357.346167:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629357.346183:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629357.346192:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629357/real 1778629357] req@ffff954342f4b100 x1865013828755200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629357.346202:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828755200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629357.346210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629357.346212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629357.346215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629357.346233:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629357.346255:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629357.346271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629357/real 1778629357] req@ffff954342f48a80 x1865013828755328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629357.346282:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828755328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629357.346288:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629357.346291:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629357.346293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629357.346298:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629357.346314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629357.346323:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629357/real 1778629357] req@ffff954342f48700 x1865013828755456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629357.346334:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828755456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629357.346341:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629357.346344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629357.346347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629357.537779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629357.537845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629358.305742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629358.561791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629358.561812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629359.329767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629359.585773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629359.585835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629360.353777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629360.609753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629360.609772:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629361.377736:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629361.633760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629361.633785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629362.401780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629362.465799:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629362.465809:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629362.465814:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629362.465823:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13696 00000100:00080000:3.0:1778629362.465860:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629362.465868:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629362.465899:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828755584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629362.465923:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629362.465926:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629362.465927:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629362.465931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13696 00000100:00080000:3.0:1778629362.465934:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629362.465938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629362.465946:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629362.465957:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828755584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778629362.465958:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828755712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629362.465970:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629362.465973:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629362.465974:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629362.465977:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13696 00000100:00080000:3.0:1778629362.465981:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629362.465984:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629362.465987:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629362.465996:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828755840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629362.466004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778629362.466013:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629362.466015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629362.466018:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629362.466029:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13696 00000100:00080000:3.0:1778629362.466033:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629362.466038:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629362.466043:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629362.466047:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629362/real 1778629362] req@ffff95434701f800 x1865013828755584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629362.466056:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828755968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629362.466062:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828755584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629362.466072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629362.466075:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778629362.466075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629362.466079:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629362.466080:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:3.0:1778629362.466086:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13696 00000100:00100000:1.0:1778629362.466087:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828755584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629362.466092:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629362.466097:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629362.466101:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629362.466109:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828756096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629362.466113:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629362.466151:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629362.466162:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629362/real 1778629362] req@ffff95434701d880 x1865013828755712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629362.466170:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828755712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629362.466175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629362.466177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629362.466179:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629362.466188:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629362.466201:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629362.466207:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629362/real 1778629362] req@ffff95434701f100 x1865013828755840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629362.466213:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828755840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629362.466218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629362.466219:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629362.466221:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629362.466239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629362.466250:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629362.466257:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629362/real 1778629362] req@ffff95434701ea00 x1865013828755968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629362.466263:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828755968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629362.466268:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629362.466271:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629362.466272:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828755968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629362.466282:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629362.466290:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629362.466294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629362/real 1778629362] req@ffff95434701c380 x1865013828756096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629362.466305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828756096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629362.466311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629362.466314:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629362.466317:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629362.657738:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629362.657760:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629363.425773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629363.681759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629363.681792:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629364.449743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629364.705722:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629364.705776:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629365.473730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629365.729817:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629365.729888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629366.497792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629366.753760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629366.754822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629367.521756:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629367.585761:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629367.585768:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629367.585772:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629367.585779:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13702 00000100:00080000:3.0:1778629367.585788:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629367.585793:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629367.585822:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828756224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629367.585861:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629367.585864:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629367.585865:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629367.585868:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13702 00000100:00080000:3.0:1778629367.585871:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629367.585873:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629367.585879:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629367.585887:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828756352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629367.585893:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629367.585895:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629367.585896:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629367.585898:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13702 00000100:00080000:3.0:1778629367.585899:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629367.585901:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629367.585903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629367.585908:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828756480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629367.585913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629367.585915:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629367.585916:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629367.585918:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13702 00000100:00080000:3.0:1778629367.585923:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629367.585926:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629367.585929:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629367.585937:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828756608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629367.585943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629367.585945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629367.585946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629367.585952:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13702 00000100:00100000:1.0:1778629367.585953:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828756224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629367.585956:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629367.585960:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629367.585963:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629367.585971:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828756736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629367.586040:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629367.586060:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629367.586072:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629367.586089:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629367.777753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629367.777776:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629368.545794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629368.801764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629368.801798:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629369.569748:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629369.825758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629369.825776:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629370.593774:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778629370.657897:0:2391:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778629370.657938:0:2391:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778629370.657953:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629370.658000:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629370.658007:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629370.658013:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629370.658023:0:2391:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778629370.658048:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629367/real 1778629370] req@ffff95434701df80 x1865013828756736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629370.658091:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828756736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629370.658102:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629370.658107:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629370.658111:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629370.658118:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629367/real 1778629370] req@ffff95434701c700 x1865013828756224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629370.658129:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828756224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629370.658136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629370.658140:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629370.658143:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828756224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629370.658149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629367/real 1778629370] req@ffff95434701c000 x1865013828756352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629370.658161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828756352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629370.658168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629370.658171:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629370.658174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629370.658179:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629367/real 1778629370] req@ffff95434701d500 x1865013828756480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629370.658203:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828756480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629370.658209:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629370.658212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629370.658215:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629370.658220:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629367/real 1778629370] req@ffff95434701f480 x1865013828756608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629370.658231:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828756608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629370.658239:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629370.658242:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629370.658245:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629370.849773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629370.849816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629371.617799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629371.873748:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629371.873808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629372.641768:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629372.705759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629372.705767:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629372.705770:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629372.705778:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13707 00000100:00080000:3.0:1778629372.705787:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629372.705794:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629372.705829:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828756864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629372.705853:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629372.705872:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629372.705874:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629372.705879:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13707 00000100:00080000:3.0:1778629372.705884:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629372.705889:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629372.705898:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629372.705903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828756864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778629372.705914:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828756992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629372.705937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629372.705940:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629372.705942:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629372.705946:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13707 00000100:00080000:3.0:1778629372.705952:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629372.705955:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629372.705961:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778629372.705977:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778629372.705977:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828757120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629372.705988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629372.705991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629372.705993:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629372.705997:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13707 00000100:00080000:3.0:1778629372.706000:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629372.706004:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629372.706017:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629372.706020:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629372/real 1778629372] req@ffff95434701d180 x1865013828756864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629372.706031:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828757248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629372.706040:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828756864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629372.706058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629372.706062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629372.706065:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629372.706073:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13707 00000100:00080000:1.0:1778629372.706080:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1778629372.706080:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629372.706087:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:1.0:1778629372.706088:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629372.706096:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828756864:192.168.204.102@tcp:250:kworker.0 00000008:00100000:3.0:1778629372.706096:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629372.706112:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828757376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629372.706129:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756992:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629372.706155:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629372.706166:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629372/real 1778629372] req@ffff95434701ca80 x1865013828756992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629372.706178:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828756992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629372.706186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629372.706189:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629372.706192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828756992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629372.706198:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757120:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629372.706215:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629372.706223:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629372/real 1778629372] req@ffff95434701e680 x1865013828757120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629372.706234:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828757120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629372.706241:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629372.706244:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629372.706247:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629372.706260:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757248:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629372.706279:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629372.706288:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629372/real 1778629372] req@ffff95434701f480 x1865013828757248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629372.706298:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828757248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629372.706305:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629372.706308:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629372.706310:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629372.706315:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757376:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629372.706339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629372.706348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629372/real 1778629372] req@ffff95434701d500 x1865013828757376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629372.706359:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828757376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629372.706366:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629372.706369:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629372.706372:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629372.897769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629372.897834:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629373.665779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629373.921804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629373.921849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629374.689769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629374.945768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629374.945814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629375.713760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629375.969774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629375.969809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629376.737791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629376.993759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629376.993808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629377.761742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629377.825748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629377.825769:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629377.825772:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629377.825779:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13712 00000100:00080000:1.0:1778629377.825788:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629377.825792:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629377.825827:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828757504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629377.825850:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629377.825852:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629377.825853:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629377.825856:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13712 00000100:00080000:1.0:1778629377.825858:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629377.825860:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629377.825867:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629377.825876:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828757632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629377.825883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629377.825885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629377.825886:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629377.825890:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828757504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629377.825891:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13712 00000100:00080000:1.0:1778629377.825896:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629377.825899:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629377.825902:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629377.825909:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828757760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629377.825920:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629377.825922:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629377.825924:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629377.825926:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13712 00000100:00080000:1.0:1778629377.825929:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629377.825932:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629377.825935:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:2.0:1778629377.825943:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629377.825944:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828757888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629377.825951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629377.825953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629377.825954:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629377.825956:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13712 00000100:00080000:1.0:1778629377.825958:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629377.825959:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629377.825961:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629377.825967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828758016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629377.825994:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629377/real 1778629377] req@ffff954350194700 x1865013828757504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629377.826018:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828757504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629377.826025:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629377.826027:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629377.826030:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828757504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629377.826046:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629377.826061:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629377.826068:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629377/real 1778629377] req@ffff954350197800 x1865013828757632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629377.826075:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828757632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629377.826086:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629377.826088:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629377.826090:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629377.826094:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629377.826103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629377.826109:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629377/real 1778629377] req@ffff954350197480 x1865013828757760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629377.826115:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828757760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629377.826127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629377.826129:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629377.826131:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629377.826134:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629377.826147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629377.826152:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629377/real 1778629377] req@ffff954350196d80 x1865013828757888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629377.826158:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828757888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629377.826163:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629377.826164:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629377.826166:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828757888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629377.826169:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629377.826177:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629377.826181:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629377/real 1778629377] req@ffff954350194000 x1865013828758016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629377.826187:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828758016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629377.826192:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629377.826193:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629377.826199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778629378.017747:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629378.017795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629378.785770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629379.041758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629379.041813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629379.809725:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629380.065753:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629380.065801:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629380.833767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629381.089770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629381.089828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629381.857777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629382.113725:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629382.113793:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629382.881763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629382.945772:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629382.945780:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629382.945784:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629382.945793:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13717 00000100:00080000:1.0:1778629382.945821:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629382.945827:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629382.945864:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828758144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629382.945894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629382.945898:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629382.945899:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629382.945904:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13717 00000100:00080000:1.0:1778629382.945907:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629382.945910:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629382.945919:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629382.945932:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828758144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629382.945934:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828758272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629382.945946:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629382.945950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629382.945951:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629382.945955:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13717 00000100:00080000:1.0:1778629382.945958:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629382.945962:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629382.945966:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629382.945977:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828758400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629382.945984:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629382.945989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629382.945997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629382.945999:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629382.946005:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13717 00000100:00080000:1.0:1778629382.946010:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629382.946014:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629382.946017:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629382/real 1778629382] req@ffff954350197100 x1865013828758144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1778629382.946020:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629382.946038:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828758528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629382.946040:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828758144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629382.946061:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629382.946065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629382.946067:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629382.946071:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13717 00000100:00080000:2.0:1778629382.946072:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629382.946075:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629382.946079:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:1.0:1778629382.946080:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629382.946085:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828758144:192.168.204.102@tcp:250:kworker.0 00000008:00100000:1.0:1778629382.946087:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629382.946099:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828758656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629382.946107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629382.946126:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629382.946135:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629382/real 1778629382] req@ffff954350194000 x1865013828758272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629382.946143:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828758272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629382.946153:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629382.946156:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629382.946158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629382.946161:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629382.946173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629382.946178:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629382/real 1778629382] req@ffff954350196d80 x1865013828758400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629382.946184:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828758400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629382.946189:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629382.946190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629382.946192:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629382.946195:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629382.946210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629382.946219:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629382/real 1778629382] req@ffff954350197480 x1865013828758528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629382.946230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828758528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629382.946237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629382.946240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629382.946248:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629382.946263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629382.946276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629382.946283:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629382/real 1778629382] req@ffff954350197800 x1865013828758656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629382.946349:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828758656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629382.946355:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629382.946357:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629382.946359:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778629383.137795:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629383.137856:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629383.905798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629384.161740:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629384.161796:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629384.929798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629385.185772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629385.185824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629385.953792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629386.209762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629386.209809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629386.977782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629387.233781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629387.233849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629388.001779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629388.065779:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629388.065788:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629388.065794:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629388.065802:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13722 00000100:00080000:2.0:1778629388.065812:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629388.065835:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629388.065872:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828758784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629388.065892:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629388.065895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629388.065896:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629388.065902:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13722 00000100:00080000:2.0:1778629388.065906:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629388.065908:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629388.065916:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629388.065924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828758912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629388.065929:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828758784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629388.065941:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629388.065943:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629388.065945:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629388.065950:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13722 00000100:00080000:2.0:1778629388.065953:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629388.065957:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629388.065962:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629388.065972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828759040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629388.065981:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629388.065984:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629388.065991:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629388.065996:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13722 00000800:00000100:1.0:1778629388.066000:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629388.066000:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629388.066015:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629388.066021:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629388.066034:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828759168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629388.066039:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629388/real 1778629388] req@ffff95435878b800 x1865013828758784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629388.066054:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629388.066058:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778629388.066061:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828758784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629388.066061:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629388.066070:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13722 00000100:00080000:2.0:1778629388.066075:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629388.066080:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629388.066088:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778629388.066091:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629388.066096:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629388.066100:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828758784:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629388.066104:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828759296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629388.066136:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758912:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629388.066159:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629388.066170:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629388/real 1778629388] req@ffff95435878a300 x1865013828758912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629388.066178:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828758912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629388.066183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629388.066185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629388.066187:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828758912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629388.066191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629388.066202:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629388.066207:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629388/real 1778629388] req@ffff95435878ad80 x1865013828759040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629388.066214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828759040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629388.066218:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629388.066220:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629388.066221:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629388.066232:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629388.066252:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629388.066262:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629388/real 1778629388] req@ffff954358789180 x1865013828759168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629388.066273:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828759168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629388.066279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629388.066281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629388.066283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629388.066286:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629388.066300:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629388.066305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629388/real 1778629388] req@ffff954358788a80 x1865013828759296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629388.066312:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828759296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629388.066316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629388.066318:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629388.066319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778629388.257770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629388.257820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629389.025740:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629389.281766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629389.281814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629390.049789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629390.305762:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629390.305819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629391.073776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629391.329751:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629391.329818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629392.097811:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629392.353771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629392.353830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629393.121782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629393.185758:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629393.185767:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629393.185771:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629393.185780:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13727 00000100:00080000:2.0:1778629393.185791:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629393.185798:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629393.185829:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828759424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629393.185857:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629393.185861:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629393.185862:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629393.185867:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13727 00000100:00080000:2.0:1778629393.185870:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629393.185874:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629393.185882:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629393.185892:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828759552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629393.185896:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828759424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629393.185908:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629393.185910:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629393.185912:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629393.185914:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13727 00000100:00080000:2.0:1778629393.185917:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629393.185920:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629393.185923:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629393.185945:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828759680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629393.185950:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629393.185959:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629393.185962:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629393.185964:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629393.185969:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13727 00000100:00080000:2.0:1778629393.185974:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629393.185978:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629393.185983:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629393.185988:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629393/real 1778629393] req@ffff954358789f80 x1865013828759424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629393.186003:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828759808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629393.186014:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629393.186017:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1778629393.186019:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828759424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629393.186019:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629393.186027:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13727 00000100:00080000:1.0:1778629393.186031:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629393.186033:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629393.186036:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778629393.186039:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629393.186041:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828759424:192.168.204.102@tcp:250:kworker.0 00000008:00100000:2.0:1778629393.186051:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629393.186061:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629393.186063:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828759936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629393.186081:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629393.186089:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629393/real 1778629393] req@ffff95435878a680 x1865013828759552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629393.186097:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828759552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629393.186102:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629393.186105:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629393.186107:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629393.186110:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629393.186119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629393.186123:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629393/real 1778629393] req@ffff954358788a80 x1865013828759680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629393.186130:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828759680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629393.186141:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629393.186143:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629393.186145:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629393.186149:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629393.186160:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629393.186165:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629393/real 1778629393] req@ffff954358789180 x1865013828759808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629393.186171:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828759808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629393.186175:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629393.186177:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629393.186178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629393.186187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629393.186195:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629393.186200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629393/real 1778629393] req@ffff95435878ad80 x1865013828759936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629393.186206:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828759936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629393.186210:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629393.186212:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629393.186217:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828759936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778629393.377735:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629393.377805:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629394.145792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629394.401782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629394.401837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629395.169758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629395.425772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629395.425816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629396.193773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629396.449732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629396.449803:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629397.217730:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629397.473752:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629397.473816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629398.241778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629398.305806:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629398.305816:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629398.305820:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629398.305829:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13732 00000100:00080000:1.0:1778629398.305840:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629398.305846:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629398.305881:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828760064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629398.305909:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629398.305913:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629398.305915:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629398.305919:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13732 00000100:00080000:1.0:1778629398.305923:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629398.305927:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629398.305936:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629398.305942:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828760064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629398.305951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828760192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629398.305962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629398.305965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629398.305967:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629398.305970:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13732 00000100:00080000:1.0:1778629398.305974:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629398.305992:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629398.305998:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1778629398.306003:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629398.306012:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828760320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629398.306022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629398.306025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629398.306027:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629398.306033:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629398/real 1778629398] req@ffff954350194700 x1865013828760064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629398.306042:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13732 00000100:00080000:1.0:1778629398.306048:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629398.306049:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828760064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629398.306056:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778629398.306060:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778629398.306062:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00080000:2.0:1778629398.306071:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629398.306077:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828760064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629398.306078:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828760448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629398.306090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629398.306093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629398.306095:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629398.306098:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760192:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778629398.306103:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13732 00000100:00080000:1.0:1778629398.306108:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629398.306113:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629398.306118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778629398.306118:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629398.306128:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629398/real 1778629398] req@ffff954350197b80 x1865013828760192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629398.306134:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828760576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629398.306140:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828760192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629398.306157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629398.306160:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629398.306163:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629398.306173:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629398.306187:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629398.306193:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629398/real 1778629398] req@ffff954350196a00 x1865013828760320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629398.306200:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828760320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629398.306205:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629398.306207:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629398.306208:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629398.306211:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629398.306222:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629398.306227:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629398/real 1778629398] req@ffff954350194e00 x1865013828760448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629398.306233:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828760448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629398.306237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629398.306239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629398.306240:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629398.306250:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629398.306259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629398.306263:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629398/real 1778629398] req@ffff954350194a80 x1865013828760576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629398.306275:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828760576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629398.306279:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629398.306281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629398.306283:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760576:192.168.204.102@tcp:8:kworker.0 00000100:00100000:3.0:1778629398.497764:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629398.497829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629399.265765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629399.522041:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629399.522871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629400.289779:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629400.545821:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629400.546253:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629401.313796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1778629401.569786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629401.569860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629402.337844:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629402.593981:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629402.593982:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629403.361810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629403.425801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629403.425839:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629403.425843:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629403.425853:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13737 00000100:00080000:1.0:1778629403.425866:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629403.425873:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629403.425912:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828760704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629403.425938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629403.425941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629403.425943:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629403.425946:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13737 00000100:00080000:1.0:1778629403.425948:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629403.425950:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629403.425959:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629403.425967:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828760832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629403.425974:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629403.425977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629403.425979:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629403.425981:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13737 00000100:00080000:1.0:1778629403.425984:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629403.425987:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629403.425989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629403.425995:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828760960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629403.426010:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629403.426012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629403.426013:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629403.426019:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13737 00000100:00080000:1.0:1778629403.426025:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629403.426026:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828760704:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629403.426031:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629403.426041:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629403.426058:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828761088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629403.426065:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629403.426069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629403.426071:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629403.426073:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13737 00000100:00080000:1.0:1778629403.426076:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629403.426079:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629403.426082:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629403.426091:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828761216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629403.426103:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629403.426141:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629403/real 1778629403] req@ffff954350195c00 x1865013828760704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629403.426155:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828760704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629403.426167:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629403.426171:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629403.426174:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828760704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629403.426179:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629403.426208:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629403.426215:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629403/real 1778629403] req@ffff954350195500 x1865013828760832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629403.426223:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828760832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629403.426228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629403.426230:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629403.426232:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760832:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629403.426251:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629403.426260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629403.426266:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629403/real 1778629403] req@ffff954350195f80 x1865013828760960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629403.426272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828760960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629403.426277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629403.426279:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629403.426280:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828760960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629403.426284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629403.426292:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629403.426297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629403/real 1778629403] req@ffff954350194380 x1865013828761088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629403.426327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828761088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629403.426333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629403.426335:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629403.426337:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629403.426341:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629403.426355:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629403.426361:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629403/real 1778629403] req@ffff954350195180 x1865013828761216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629403.426367:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828761216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629403.426373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629403.426375:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629403.426376:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761216:192.168.204.102@tcp:8:kworker.0 00000100:00100000:0.0:1778629403.619265:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629403.619535:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629404.385751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778629404.641812:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629404.641926:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629405.409787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629405.665783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629405.665842:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629406.433763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629406.690149:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629406.690525:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629407.457742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629407.713782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629407.713832:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629408.481778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629408.545760:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629408.545792:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629408.545796:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629408.545804:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13743 00000100:00080000:1.0:1778629408.545815:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629408.545821:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629408.545852:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828761344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629408.545876:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629408.545879:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629408.545880:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629408.545883:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13743 00000100:00080000:1.0:1778629408.545886:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629408.545888:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629408.545896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629408.545907:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828761472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629408.545915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629408.545915:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828761344:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629408.545918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629408.545919:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629408.545922:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13743 00000100:00080000:1.0:1778629408.545925:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629408.545930:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629408.545933:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629408.545940:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828761600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629408.545951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629408.545953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629408.545954:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629408.545957:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13743 00000100:00080000:1.0:1778629408.545961:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629408.545963:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629408.545966:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629408.545973:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828761728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629408.545979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629408.545981:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629408.545982:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778629408.545986:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629408.545987:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13743 00000100:00080000:1.0:1778629408.545991:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629408.545993:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629408.545996:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629408.546003:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828761856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629408.546022:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629408/real 1778629408] req@ffff954350195880 x1865013828761344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629408.546047:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828761344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629408.546057:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629408.546061:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629408.546065:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828761344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629408.546086:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629408.546106:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629408.546116:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629408/real 1778629408] req@ffff954350195180 x1865013828761472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629408.546128:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828761472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629408.546136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629408.546139:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629408.546142:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629408.546146:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629408.546161:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629408.546169:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629408/real 1778629408] req@ffff954350194380 x1865013828761600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629408.546179:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828761600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629408.546197:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629408.546200:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629408.546203:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629408.546220:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629408.546246:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629408.546255:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629408/real 1778629408] req@ffff954350195f80 x1865013828761728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629408.546265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828761728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629408.546274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629408.546276:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629408.546279:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629408.546284:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629408.546297:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629408.546304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629408/real 1778629408] req@ffff954350195500 x1865013828761856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629408.546315:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828761856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629408.546322:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629408.546325:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629408.546332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828761856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629408.737773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629408.737836:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629409.505780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629409.761782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629409.761917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629410.529760:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629410.785776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629410.785852:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629411.553771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629411.809776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629411.809822:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629412.577745:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629412.833765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629412.833824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629413.601782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629413.665809:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629413.665816:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629413.665820:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629413.665827:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13748 00000100:00080000:0.0:1778629413.665836:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629413.665841:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629413.665869:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49f80 x1865013828761984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629413.665888:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629413.665891:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629413.665893:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629413.665897:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13748 00000100:00080000:0.0:1778629413.665901:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629413.665904:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629413.665912:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629413.665924:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a680 x1865013828762112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629413.665925:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828761984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629413.665933:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629413.665936:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629413.665949:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629413.665952:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13748 00000100:00080000:0.0:1778629413.665955:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629413.665958:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629413.665962:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629413.665971:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48700 x1865013828762240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1778629413.665977:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629413.665979:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629413.665981:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629413.665982:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629413.665986:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13748 00000100:00080000:0.0:1778629413.665988:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629413.665991:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629413.665995:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629413.666002:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48a80 x1865013828762368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629413.666007:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629413/real 1778629413] req@ffff954342f49f80 x1865013828761984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629413.666010:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629413.666013:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629413.666015:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629413.666017:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13748 00000100:00080000:0.0:1778629413.666020:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629413.666023:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629413.666026:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629413.666027:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49f80 x1865013828761984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629413.666036:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778629413.666039:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b100 x1865013828762496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629413.666040:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629413.666043:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828761984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629413.666064:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629413.666085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629413.666095:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629413/real 1778629413] req@ffff954342f4a680 x1865013828762112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629413.666102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a680 x1865013828762112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629413.666107:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629413.666122:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629413.666129:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629413.666133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629413.666147:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629413.666153:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629413/real 1778629413] req@ffff954342f48700 x1865013828762240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629413.666161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48700 x1865013828762240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629413.666165:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629413.666167:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629413.666169:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629413.666172:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629413.666181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629413.666185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629413/real 1778629413] req@ffff954342f48a80 x1865013828762368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629413.666191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48a80 x1865013828762368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629413.666195:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629413.666197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629413.666199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629413.666201:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629413.666209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629413.666218:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629413/real 1778629413] req@ffff954342f4b100 x1865013828762496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629413.666225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b100 x1865013828762496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629413.666230:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629413.666231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629413.666233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629413.857782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629413.857849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629414.625783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629414.881754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629414.881813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629415.649818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629415.905887:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629415.907231:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629416.673800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629416.929732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629416.929862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629417.697826:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629417.953823:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629417.953888:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629418.721743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629418.785807:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629418.785817:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629418.785822:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629418.785832:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13753 00000100:00080000:2.0:1778629418.785843:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629418.785849:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629418.785891:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828762624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629418.785919:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629418.785923:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629418.785925:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629418.785929:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13753 00000100:00080000:2.0:1778629418.785933:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629418.785957:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629418.785967:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629418.785980:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828762752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629418.785991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629418.785995:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629418.785997:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778629418.786006:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828762624:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629418.786006:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13753 00000100:00080000:2.0:1778629418.786033:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629418.786038:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629418.786044:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629418.786054:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828762880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629418.786064:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629418.786067:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629418.786070:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629418.786074:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13753 00000100:00080000:2.0:1778629418.786078:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629418.786083:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629418.786096:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629418.786107:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828763008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629418.786119:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629418.786121:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629418.786131:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629418.786134:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629418.786140:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13753 00000100:00080000:2.0:1778629418.786146:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629418.786151:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629418.786162:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629418.786174:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629418/real 1778629418] req@ffff95435878a300 x1865013828762624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629418.786179:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828763136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629418.786218:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828762624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629418.786232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629418.786238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629418.786242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828762624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629418.786271:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629418.786296:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629418.786308:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629418/real 1778629418] req@ffff95435878b800 x1865013828762752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629418.786319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828762752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629418.786326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629418.786329:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629418.786331:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629418.786353:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629418.786374:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629418.786384:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629418/real 1778629418] req@ffff954358789500 x1865013828762880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629418.786394:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828762880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629418.786401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629418.786404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629418.786407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828762880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629418.786413:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629418.786431:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629418.786440:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629418/real 1778629418] req@ffff954358789880 x1865013828763008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629418.786458:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828763008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629418.786466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629418.786469:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629418.786472:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629418.786478:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629418.786497:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629418.786506:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629418/real 1778629418] req@ffff954358788700 x1865013828763136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629418.786525:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828763136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629418.786534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629418.786537:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629418.786540:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629418.977759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629418.977808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629419.745773:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629420.001770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629420.001816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629420.769797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629421.025755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629421.025820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629421.793766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629422.049758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629422.049808:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629422.817772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629423.073734:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629423.073785:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629423.841746:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629423.905763:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629423.905770:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629423.905774:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629423.905781:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13758 00000100:00080000:3.0:1778629423.905790:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629423.905795:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629423.905828:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828763264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629423.905853:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629423.905857:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629423.905859:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629423.905862:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13758 00000100:00080000:3.0:1778629423.905866:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629423.905869:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629423.905877:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629423.905887:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828763392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629423.905891:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828763264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629423.905918:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629423.905921:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629423.905922:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629423.905925:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13758 00000100:00080000:3.0:1778629423.905929:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629423.905931:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629423.905935:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629423.905943:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828763520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629423.905950:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629423.905952:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629423.905953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629423.905957:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13758 00000800:00000100:1.0:1778629423.905958:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629423.905960:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629423.905963:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629423.905966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629423.905975:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828763648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629423.905981:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629423.905983:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629423.905984:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778629423.905988:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629423/real 1778629423] req@ffff95434701c000 x1865013828763264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629423.905989:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13758 00000100:00080000:3.0:1778629423.905998:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629423.906002:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629423.906006:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828763264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778629423.906006:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778629423.906015:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778629423.906018:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828763776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629423.906021:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629423.906027:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828763264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629423.906059:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629423.906082:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629423.906101:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629423/real 1778629423] req@ffff95434701c700 x1865013828763392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629423.906125:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828763392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629423.906133:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629423.906136:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629423.906139:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629423.906144:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629423.906162:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629423.906172:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629423/real 1778629423] req@ffff95434701df80 x1865013828763520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629423.906183:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828763520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629423.906190:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629423.906194:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629423.906197:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629423.906202:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629423.906220:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629423.906230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629423/real 1778629423] req@ffff95434701c380 x1865013828763648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629423.906241:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828763648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629423.906258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629423.906262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629423.906265:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629423.906270:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629423.906289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629423.906297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629423/real 1778629423] req@ffff95434701ea00 x1865013828763776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629423.906308:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828763776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629423.906315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629423.906318:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629423.906321:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828763776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629424.097778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629424.097846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629424.865788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629425.121769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629425.121829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629425.889777:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629426.145771:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629426.145819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629426.913770:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629427.169766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629427.169821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629427.937753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629428.193792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629428.193840:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629428.961766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629429.025766:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629429.025774:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629429.025778:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629429.025787:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13763 00000100:00080000:1.0:1778629429.025798:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629429.025820:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629429.025867:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828763904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629429.025895:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629429.025899:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629429.025900:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629429.025905:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13763 00000100:00080000:1.0:1778629429.025909:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629429.025912:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629429.025922:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629429.025934:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828763904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629429.025936:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828764032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629429.025951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629429.025954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629429.025956:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629429.025959:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13763 00000100:00080000:1.0:1778629429.025962:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629429.025966:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629429.025970:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629429.025980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828764160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629429.025990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629429.025993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629429.025995:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1778629429.026000:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629429.026008:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13763 00000100:00080000:1.0:1778629429.026014:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629429.026018:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629429.026023:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629429.026035:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828764288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629429.026039:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629429/real 1778629429] req@ffff954350195c00 x1865013828763904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629429.026054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629429.026060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629429.026061:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629429.026066:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13763 00000100:00100000:2.0:1778629429.026070:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828763904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629429.026071:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629429.026080:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778629429.026082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1778629429.026084:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:2.0:1778629429.026093:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629429.026098:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828764416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629429.026101:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828763904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629429.026133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629429.026156:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629429.026166:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629429/real 1778629429] req@ffff954350194a80 x1865013828764032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629429.026174:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828764032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629429.026179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629429.026181:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629429.026183:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629429.026187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629429.026198:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629429.026203:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629429/real 1778629429] req@ffff954350194e00 x1865013828764160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629429.026210:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828764160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629429.026214:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629429.026216:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629429.026218:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629429.026221:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629429.026237:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629429.026243:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629429/real 1778629429] req@ffff954350196a00 x1865013828764288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629429.026251:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828764288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629429.026258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629429.026261:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629429.026264:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629429.026269:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629429.026287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629429.026297:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629429/real 1778629429] req@ffff954350197b80 x1865013828764416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629429.026307:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828764416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629429.026315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629429.026317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629429.026320:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764416:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629429.217779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629429.217862:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629429.985762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629430.241769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629430.241813:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629431.009823:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629431.265780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629431.265824:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629432.033793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629432.289829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629432.289965:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629433.057843:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629433.313786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629433.313871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629434.081775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629434.145786:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629434.145794:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629434.145799:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629434.145825:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13768 00000100:00080000:1.0:1778629434.145836:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629434.145842:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629434.145872:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828764544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629434.145900:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629434.145904:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629434.145906:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629434.145911:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13768 00000100:00080000:1.0:1778629434.145914:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629434.145918:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629434.145926:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629434.145939:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828764672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629434.145944:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828764544:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629434.145954:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629434.145958:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629434.145960:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629434.145964:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13768 00000100:00080000:1.0:1778629434.145967:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629434.145978:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629434.145983:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629434.145992:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828764800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629434.146015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629434.146021:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629434.146023:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629434.146028:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13768 00000100:00080000:1.0:1778629434.146032:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629434.146036:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629434.146041:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764672:192.168.204.102@tcp:38:kworker.0 00000008:00100000:1.0:1778629434.146042:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629434.146058:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828764928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629434.146070:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629434.146074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629434.146075:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629434.146079:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13768 00000100:00080000:1.0:1778629434.146083:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1778629434.146084:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764800:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778629434.146091:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629434.146097:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629434.146107:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629434.146112:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828765056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629434.146133:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629434.337768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629434.337829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629435.105783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629435.361770:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629435.361835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629436.129766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629436.385728:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629436.385797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629437.153778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1778629437.218037:0:2389:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:3.0:1778629437.218111:0:2389:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1778629437.218133:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629437.218182:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629437.218193:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629437.218203:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:3.0:1778629437.218213:0:2389:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:2.0:1778629437.218238:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629434/real 1778629437] req@ffff954350194000 x1865013828765056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629437.218272:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828765056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629437.218286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629437.218292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629437.218297:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629437.218305:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629434/real 1778629437] req@ffff954350196d80 x1865013828764928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629437.218327:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828764928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629437.218333:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629437.218336:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629437.218339:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629437.218345:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629434/real 1778629437] req@ffff954350197800 x1865013828764672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629437.218381:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828764672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629437.218389:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629437.218393:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629437.218395:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629437.218401:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629434/real 1778629437] req@ffff954350197480 x1865013828764800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629437.218412:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828764800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629437.218418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629437.218422:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629437.218425:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828764800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629437.218431:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629434/real 1778629437] req@ffff954350194700 x1865013828764544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629437.218441:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828764544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629437.218453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629437.218455:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629437.218458:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828764544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629437.409804:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629437.409872:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629438.177785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629438.433749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629438.433809:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629439.201769:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629439.265789:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629439.265797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629439.265800:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629439.265807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13773 00000100:00080000:2.0:1778629439.265816:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629439.265820:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629439.265846:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828765184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629439.265873:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629439.265876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629439.265877:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629439.265881:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13773 00000100:00080000:2.0:1778629439.265884:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629439.265887:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629439.265896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629439.265913:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828765184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629439.265915:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828765312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629439.265931:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629439.265935:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629439.265937:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629439.265941:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13773 00000100:00080000:2.0:1778629439.265944:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629439.265948:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629439.265952:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1778629439.265977:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629439.265980:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828765440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629439.265991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629439.265994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629439.265996:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629439.266000:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13773 00000100:00080000:2.0:1778629439.266006:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629439.266010:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629439/real 1778629439] req@ffff954358788e00 x1865013828765184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629439.266012:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629439.266018:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629439.266035:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828765184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629439.266040:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828765568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629439.266049:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629439.266058:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:2.0:1778629439.266063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629439.266065:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828765184:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629439.266070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629439.266073:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629439.266078:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13773 00000100:00080000:2.0:1778629439.266083:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629439.266086:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629439.266092:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629439.266094:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629439.266108:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828765696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629439.266120:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629439.266133:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629439/real 1778629439] req@ffff954358789c00 x1865013828765312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629439.266145:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828765312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629439.266153:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629439.266156:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629439.266158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629439.266164:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765440:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629439.266179:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629439.266186:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629439/real 1778629439] req@ffff954358788380 x1865013828765440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629439.266207:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828765440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629439.266215:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629439.266224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629439.266227:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629439.266241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765568:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629439.266260:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629439.266269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629439/real 1778629439] req@ffff95435878aa00 x1865013828765568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629439.266279:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828765568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629439.266287:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629439.266289:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629439.266292:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629439.266306:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765696:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629439.266321:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629439.266329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629439/real 1778629439] req@ffff95435878b100 x1865013828765696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629439.266339:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828765696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629439.266357:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629439.266360:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629439.266362:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629439.457767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629439.457828:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629440.225795:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629440.481766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629440.481837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629441.249776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629441.505767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629441.505835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629442.273776:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629442.529754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629442.529816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629443.297790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629443.553775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629443.553827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629444.321784:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629444.385760:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629444.385770:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629444.385774:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629444.385798:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13778 00000100:00080000:2.0:1778629444.385808:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629444.385812:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629444.385838:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828765824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629444.385859:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629444.385862:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629444.385863:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629444.385865:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13778 00000100:00080000:2.0:1778629444.385867:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629444.385869:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629444.385876:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629444.385882:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828765952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629444.385888:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629444.385890:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629444.385891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629444.385894:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13778 00000100:00080000:2.0:1778629444.385896:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629444.385898:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629444.385900:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629444.385906:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828766080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629444.385913:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828765824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629444.385926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629444.385928:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629444.385929:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629444.385933:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13778 00000100:00080000:2.0:1778629444.385936:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629444.385938:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629444.385941:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629444.385950:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828766208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629444.385955:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629444.385957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629444.385958:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629444.385960:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13778 00000100:00080000:2.0:1778629444.385962:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629444.385964:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629444.385966:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629444.385972:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828766336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629444.385977:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629444.386010:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629444/real 1778629444] req@ffff95435878bb80 x1865013828765824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629444.386025:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828765824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629444.386034:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629444.386038:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629444.386041:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828765824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629444.386048:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629444.386066:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629444.386075:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629444/real 1778629444] req@ffff95435878b480 x1865013828765952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629444.386085:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828765952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629444.386098:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629444.386102:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629444.386105:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828765952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629444.386127:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629444.386145:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629444.386154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629444/real 1778629444] req@ffff95435878b100 x1865013828766080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629444.386164:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828766080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629444.386171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629444.386174:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629444.386177:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629444.386182:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629444.386196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629444.386204:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629444/real 1778629444] req@ffff95435878aa00 x1865013828766208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629444.386214:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828766208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629444.386221:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629444.386224:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629444.386231:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629444.386237:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629444.386253:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629444.386261:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629444/real 1778629444] req@ffff954358788380 x1865013828766336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629444.386271:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828766336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629444.386278:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629444.386281:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629444.386284:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766336:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629444.577765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629444.577819:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629445.345763:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629445.601760:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629445.601814:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629446.369772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629446.625767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629446.625846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629447.393771:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629447.649769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629447.649844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629448.417751:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629448.673768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629448.673827:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629449.441747:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629449.505831:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629449.505840:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629449.505845:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629449.505854:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13784 00000100:00080000:3.0:1778629449.505888:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629449.505894:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629449.505929:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828766464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629449.505953:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629449.505955:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629449.505957:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629449.505960:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13784 00000100:00080000:3.0:1778629449.505962:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629449.505964:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629449.505973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629449.505981:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828766592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629449.505988:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629449.505991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629449.505992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629449.505995:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13784 00000100:00080000:3.0:1778629449.505996:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629449.505999:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629449.506002:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629449.506010:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828766464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:3.0:1778629449.506011:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828766720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629449.506032:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629449.506042:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629449.506044:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629449.506050:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13784 00000100:00080000:3.0:1778629449.506055:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629449.506060:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629449.506067:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629449.506082:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828766848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629449.506092:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778629449.506095:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629449.506096:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629449.506101:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629449.506105:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13784 00000100:00080000:3.0:1778629449.506110:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629449.506113:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629449.506118:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629449.506127:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828766976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629449.506140:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629449/real 1778629449] req@ffff95434701f100 x1865013828766464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629449.506161:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828766464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629449.506172:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629449.506176:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629449.506197:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828766464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629449.506205:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629449.506230:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629449.506242:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629449/real 1778629449] req@ffff95434701d880 x1865013828766592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629449.506254:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828766592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629449.506262:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629449.506264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629449.506266:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629449.506285:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629449.506298:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629449.506304:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629449/real 1778629449] req@ffff95434701f800 x1865013828766720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629449.506311:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828766720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629449.506315:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629449.506317:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629449.506319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629449.506322:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629449.506331:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629449.506347:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629449/real 1778629449] req@ffff95434701ea00 x1865013828766848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629449.506357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828766848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629449.506365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629449.506368:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629449.506371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629449.506376:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629449.506394:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629449.506415:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629449/real 1778629449] req@ffff95434701c380 x1865013828766976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629449.506427:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828766976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629449.506435:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629449.506438:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629449.506441:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828766976:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629449.697784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629449.697870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629450.465780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629450.721822:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629450.721894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629451.489758:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629451.745788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629451.745838:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629452.513781:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629452.769767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629452.769821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629453.537742:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629453.793781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629453.793869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629454.561783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629454.625761:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629454.625768:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629454.625785:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629454.625792:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13789 00000100:00080000:3.0:1778629454.625802:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629454.625807:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629454.625833:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828767104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629454.625854:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629454.625856:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629454.625857:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629454.625860:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13789 00000100:00080000:3.0:1778629454.625862:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629454.625864:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629454.625870:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629454.625877:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828767232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629454.625886:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629454.625889:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629454.625891:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629454.625895:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13789 00000100:00080000:3.0:1778629454.625898:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629454.625903:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828767104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629454.625904:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629454.625910:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629454.625921:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828767360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629454.625943:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629454.625946:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629454.625948:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629454.625951:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13789 00000100:00080000:3.0:1778629454.625956:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629454.625960:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629454.625965:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629454.625966:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:3.0:1778629454.625979:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828767488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629454.625988:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629454.625991:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629454.625992:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629454.625995:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13789 00000100:00080000:3.0:1778629454.625999:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629454.626001:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629454.626005:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629454/real 1778629454] req@ffff95434701df80 x1865013828767104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1778629454.626005:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629454.626017:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828767616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629454.626032:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828767104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629454.626042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629454.626047:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629454.626051:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828767104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629454.626058:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767232:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629454.626080:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629454.626091:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629454/real 1778629454] req@ffff95434701c700 x1865013828767232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629454.626102:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828767232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629454.626108:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629454.626111:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629454.626113:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629454.626131:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629454.626143:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629454.626149:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629454/real 1778629454] req@ffff95434701c000 x1865013828767360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629454.626156:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828767360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629454.626170:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629454.626174:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629454.626177:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629454.626182:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629454.626200:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629454.626210:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629454/real 1778629454] req@ffff95434701d500 x1865013828767488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629454.626221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828767488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629454.626235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629454.626238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629454.626241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629454.626246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629454.626264:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629454.626273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629454/real 1778629454] req@ffff95434701f480 x1865013828767616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629454.626280:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828767616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629454.626285:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629454.626287:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629454.626293:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767616:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629454.817779:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629454.817841:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629455.585775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629455.841781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629455.841903:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629456.609782:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629456.865782:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629456.865875:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629457.633743:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629457.889732:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629457.889864:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629458.657767:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629458.913793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629458.913860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629459.681764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629459.745801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629459.745812:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629459.745816:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629459.745826:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13794 00000100:00080000:2.0:1778629459.745854:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629459.745860:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629459.745892:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828767744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629459.745915:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629459.745918:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629459.745919:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629459.745922:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13794 00000100:00080000:2.0:1778629459.745925:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629459.745927:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629459.745942:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629459.745953:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828767872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629459.745961:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828767744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629459.745962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629459.745969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629459.745971:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629459.745975:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13794 00000100:00080000:2.0:1778629459.745977:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629459.745979:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629459.745981:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629459.745989:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828768000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629459.745995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629459.746001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629459.746003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629459.746006:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13794 00000100:00080000:2.0:1778629459.746010:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629459.746014:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629459.746026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629459.746028:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629459.746040:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828768128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629459.746050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629459.746053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629459.746055:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629459.746059:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13794 00000100:00080000:2.0:1778629459.746063:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629459.746067:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629459.746071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629459.746076:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629459/real 1778629459] req@ffff954358789c00 x1865013828767744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629459.746083:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828768256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629459.746104:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828767744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629459.746115:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629459.746119:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629459.746123:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828767744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629459.746147:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629459.746178:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629459.746190:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629459/real 1778629459] req@ffff954358788e00 x1865013828767872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629459.746212:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828767872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629459.746220:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629459.746223:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629459.746234:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828767872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629459.746240:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629459.746259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629459.746269:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629459/real 1778629459] req@ffff954358788700 x1865013828768000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629459.746279:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828768000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629459.746286:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629459.746288:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629459.746291:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629459.746296:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629459.746314:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629459.746322:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629459/real 1778629459] req@ffff954358789880 x1865013828768128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629459.746332:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828768128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629459.746338:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629459.746340:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629459.746343:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629459.746348:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629459.746363:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629459.746379:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629459/real 1778629459] req@ffff954358789500 x1865013828768256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629459.746388:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828768256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629459.746396:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629459.746398:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629459.746401:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629459.937754:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629459.937816:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629460.705797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629460.961813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629460.961940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629461.729860:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629461.986077:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629461.987323:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629462.753853:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629463.009819:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629463.009915:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629463.778041:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629464.033808:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629464.033917:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629464.801857:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629464.865904:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629464.865919:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629464.865924:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629464.865995:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13799 00000100:00080000:3.0:1778629464.866012:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629464.866019:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629464.866072:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828768384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629464.866175:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828768384:192.168.204.102@tcp:250:kworker.0 00000800:00000100:1.0:1778629464.866259:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629464.866528:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629464.866533:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629464.866535:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629464.866541:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13799 00000100:00080000:3.0:1778629464.866547:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629464.866552:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629464.866563:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629464.866578:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828768512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629464.867008:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629464/real 1778629464] req@ffff95434701e680 x1865013828768384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629464.867031:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828768384/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629464.867042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629464.867046:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629464.867050:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828768384:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629464.867076:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768512:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629464.867113:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629464.867150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629464/real 1778629464] req@ffff95434701ca80 x1865013828768512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629464.867163:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629464.867164:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828768512/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629464.867173:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629464.867179:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629464.867180:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629464.867185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629464.867190:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768512:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778629464.867195:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13799 00000100:00080000:3.0:1778629464.867201:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629464.867206:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629464.867217:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629464.867232:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828768640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629464.867376:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768640:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778629464.867377:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629464.867383:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629464.867385:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629464.867391:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13799 00000100:00080000:3.0:1778629464.867396:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629464.867400:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629464.867406:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629464.867419:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828768768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629464.867429:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629464.867432:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629464.867434:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629464.867438:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13799 00000100:00080000:3.0:1778629464.867441:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629464.867445:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629464.867449:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629464.867462:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828768896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629464.867477:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629464.867549:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629464/real 1778629464] req@ffff95434701d180 x1865013828768640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629464.867561:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828768640/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629464.867568:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629464.867571:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629464.867574:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768640:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629464.867586:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768768:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629464.867605:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629464.867614:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629464/real 1778629464] req@ffff95434701ca80 x1865013828768768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629464.867624:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828768768/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629464.867635:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629464.867639:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629464.867641:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768768:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629464.867646:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768896:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629464.867662:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629464.867685:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629464/real 1778629464] req@ffff95434701e680 x1865013828768896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629464.867696:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828768896/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629464.867703:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629464.867706:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629464.867708:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828768896:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629465.058183:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629465.058263:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629465.825810:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629466.081801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629466.081948:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629466.850305:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629467.107252:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629467.108108:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629467.873868:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629468.131351:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629468.131712:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629468.897794:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629469.153813:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629469.154061:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629469.921799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629469.985817:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629469.985826:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629469.985830:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629469.985840:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13804 00000100:00080000:3.0:1778629469.985852:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629469.985858:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629469.985894:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828769024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629469.985973:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828769024:192.168.204.102@tcp:250:kworker.0 00000800:00000100:1.0:1778629469.986045:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629469.986075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629469.986079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629469.986081:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629469.986085:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13804 00000100:00080000:3.0:1778629469.986089:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629469.986093:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629469.986102:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629469.986115:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828769152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629469.986271:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629469/real 1778629469] req@ffff95434701f480 x1865013828769024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629469.986289:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828769024/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629469.986316:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629469.986320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629469.986323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828769024:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629469.986347:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769152:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629469.986369:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629469.986380:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629469/real 1778629469] req@ffff95434701d500 x1865013828769152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629469.986391:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828769152/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629469.986399:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629469.986402:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629469.986405:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769152:192.168.204.102@tcp:38:kworker.0 00000100:00080000:3.0:1778629469.986693:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629469.986698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629469.986706:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629469.986710:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13804 00000100:00080000:3.0:1778629469.986714:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629469.986718:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629469.986724:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629469.986736:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828769280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629469.986780:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769280:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629469.986803:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629469.986992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629469.987005:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629469.987008:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629469.987012:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13804 00000100:00080000:3.0:1778629469.987017:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629469.987021:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629469.987026:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629469.987039:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828769408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629469.987078:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629469/real 1778629469] req@ffff95434701c000 x1865013828769280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629469.987089:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828769280/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629469.987096:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629469.987100:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629469.987103:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769280:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629469.987121:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769408:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629469.987148:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629469.987158:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629469/real 1778629469] req@ffff95434701c700 x1865013828769408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629469.987172:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828769408/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629469.987179:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629469.987181:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629469.987184:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769408:192.168.204.102@tcp:8:kworker.0 00000100:00080000:3.0:1778629469.987264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629469.987268:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629469.987269:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629469.987273:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13804 00000100:00080000:3.0:1778629469.987277:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629469.987281:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629469.987286:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629469.987304:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828769536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629469.987569:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769536:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629469.987592:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629469.987642:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629469/real 1778629469] req@ffff95434701df80 x1865013828769536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629469.987650:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828769536/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629469.987655:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629469.987657:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629469.987659:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769536:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629470.177861:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629470.177942:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629470.946053:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629471.201856:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629471.202146:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629471.969789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629472.225824:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629472.226395:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629472.993790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629473.249784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629473.250037:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629474.018071:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629474.273776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629474.273876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629475.041799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629475.105857:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629475.105867:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629475.105871:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629475.105882:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13809 00000100:00080000:3.0:1778629475.105893:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629475.105915:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629475.105951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828769664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629475.105983:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629475.105987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629475.105989:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629475.105992:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13809 00000100:00080000:3.0:1778629475.105996:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629475.105999:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629475.106007:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629475.106019:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828769792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629475.106029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629475.106043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629475.106045:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629475.106049:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13809 00000100:00080000:3.0:1778629475.106053:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629475.106056:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629475.106061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629475.106344:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828769664:192.168.204.102@tcp:250:kworker.0 00000800:00000100:1.0:1778629475.106399:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629475.106429:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629475/real 1778629475] req@ffff95434701c380 x1865013828769664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629475.106445:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828769664/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629475.106453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629475.106457:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629475.106461:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828769664:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629475.106467:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769792:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629475.106488:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629475.106500:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629475/real 1778629475] req@ffff95434701ea00 x1865013828769792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629475.106511:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828769792/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629475.106518:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629475.106521:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629475.106525:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769792:192.168.204.102@tcp:38:kworker.0 00000100:00100000:3.0:1778629475.106828:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828769920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629475.106847:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629475.106851:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629475.106853:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629475.106857:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13809 00000100:00080000:3.0:1778629475.106862:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629475.106866:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629475.106871:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629475.106882:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828770048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629475.106889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629475.106893:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629475.106894:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629475.106909:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13809 00000100:00080000:3.0:1778629475.106912:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629475.106915:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629475.106926:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629475.106937:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828770176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629475.106994:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769920:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629475.107017:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629475.107028:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629475/real 1778629475] req@ffff95434701f800 x1865013828769920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629475.107046:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828769920/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629475.107054:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629475.107058:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629475.107061:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828769920:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629475.107313:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770048:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629475.107337:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629475.107348:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629475/real 1778629475] req@ffff95434701d880 x1865013828770048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629475.107360:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828770048/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629475.107368:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629475.107371:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629475.107374:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770048:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629475.107379:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770176:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629475.107396:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629475.107404:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629475/real 1778629475] req@ffff95434701f100 x1865013828770176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629475.107414:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828770176/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629475.107435:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629475.107438:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629475.107441:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770176:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629475.297775:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629475.298395:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629476.065783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629476.321863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629476.323869:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629477.089872:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629477.345772:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629477.345835:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629478.114031:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629478.369777:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629478.369966:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629479.137956:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629479.393797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629479.393874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629480.161820:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629480.225894:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629480.225905:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629480.225909:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629480.225918:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13814 00000100:00080000:2.0:1778629480.225928:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629480.225935:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629480.225980:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828770304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629480.226028:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629480.226032:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629480.226034:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629480.226038:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13814 00000100:00080000:2.0:1778629480.226043:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629480.226046:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629480.226055:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629480.226069:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828770432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629480.226080:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828770304:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629480.226080:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629480.226086:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629480.226089:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629480.226093:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13814 00000100:00080000:2.0:1778629480.226096:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629480.226099:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629480.226103:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629480.226126:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828770560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629480.226135:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629480.226138:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629480.226139:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629480.226143:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13814 00000100:00080000:2.0:1778629480.226152:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629480.226157:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629480.226163:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:2.0:1778629480.226164:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629480.226177:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828770688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629480.226187:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629480.226189:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629480.226192:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629480.226195:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13814 00000100:00100000:1.0:1778629480.226200:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629480/real 1778629480] req@ffff95435878b800 x1865013828770304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629480.226200:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629480.226206:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629480.226214:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629480.226228:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828770304/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629480.226228:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828770816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629480.226258:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629480.226264:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629480.226268:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828770304:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629480.226291:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770432:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629480.226311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629480.226321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629480/real 1778629480] req@ffff95435878a300 x1865013828770432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629480.226332:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828770432/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629480.226351:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629480.226354:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629480.226357:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770432:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629480.226362:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770560:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629480.226378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629480.226387:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629480/real 1778629480] req@ffff95435878ad80 x1865013828770560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629480.226397:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828770560/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629480.226404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629480.226407:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629480.226410:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770560:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629480.226415:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770688:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629480.226430:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629480.226437:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629480/real 1778629480] req@ffff954358789180 x1865013828770688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629480.226447:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828770688/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629480.226453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629480.226456:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629480.226464:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770688:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629480.226470:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770816:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629480.226489:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629480.226508:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629480/real 1778629480] req@ffff954358788a80 x1865013828770816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629480.226518:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828770816/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629480.226525:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629480.226528:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629480.226531:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828770816:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629480.417781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629480.418113:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629481.185805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629481.441796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629481.441975:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629482.209817:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629482.465870:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629482.466054:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629483.233846:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629483.489873:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629483.489874:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629484.257808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629484.513759:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629484.513829:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629485.281863:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629485.345849:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629485.345869:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629485.345874:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629485.345887:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13819 00000100:00080000:2.0:1778629485.345901:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629485.345909:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629485.345965:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a680 x1865013828770944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629485.345999:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629485.346021:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629485.346023:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629485.346029:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13819 00000100:00080000:2.0:1778629485.346034:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629485.346038:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629485.346050:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629485.346066:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789f80 x1865013828771072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629485.346077:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629485.346081:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629485.346084:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629485.346094:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13819 00000100:00100000:1.0:1778629485.346095:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828770944:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629485.346099:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629485.346108:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629485.346114:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629485.346127:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788a80 x1865013828771200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629485.346136:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629485.346140:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629485.346142:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629485.346146:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13819 00000100:00080000:2.0:1778629485.346150:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629485.346154:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629485.346167:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000800:00000100:1.0:1778629485.346182:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629485.346183:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789180 x1865013828771328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629485.346198:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629485.346202:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629485.346204:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629485.346210:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13819 00000100:00080000:2.0:1778629485.346216:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629485.346230:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629485/real 1778629485] req@ffff95435878a680 x1865013828770944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629485.346238:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629485.346251:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629485.346265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a680 x1865013828770944/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629485.346266:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878ad80 x1865013828771456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629485.346309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629485.346318:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629485.346338:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828770944:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629485.346367:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771072:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629485.346393:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629485.346405:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629485/real 1778629485] req@ffff954358789f80 x1865013828771072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629485.346417:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789f80 x1865013828771072/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629485.346426:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629485.346429:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629485.346433:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771072:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629485.346439:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771200:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629485.346456:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629485.346465:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629485/real 1778629485] req@ffff954358788a80 x1865013828771200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629485.346477:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788a80 x1865013828771200/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629485.346485:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629485.346488:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629485.346491:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771200:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629485.346497:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771328:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629485.346514:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629485.346530:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629485/real 1778629485] req@ffff954358789180 x1865013828771328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629485.346542:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789180 x1865013828771328/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629485.346549:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629485.346553:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629485.346556:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771328:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629485.346562:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771456:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629485.346579:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629485.346588:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629485/real 1778629485] req@ffff95435878ad80 x1865013828771456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629485.346599:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878ad80 x1865013828771456/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629485.346607:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629485.346610:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629485.346613:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771456:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629485.537837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629485.537931:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629486.306194:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629486.562003:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629486.562034:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629487.329803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629487.585880:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629487.587772:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629488.353762:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629488.609773:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629488.609825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629489.377799:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629489.633863:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629489.633903:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629490.401731:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629490.465780:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629490.465787:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629490.465791:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629490.465798:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13824 00000100:00080000:3.0:1778629490.465807:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629490.465811:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629490.465843:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828771584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629490.465862:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629490.465865:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629490.465877:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629490.465881:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13824 00000100:00080000:3.0:1778629490.465883:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629490.465886:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629490.465894:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629490.465903:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828771712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629490.465910:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629490.465913:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828771584:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629490.465913:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629490.465915:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629490.465921:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13824 00000100:00080000:3.0:1778629490.465923:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629490.465925:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629490.465927:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629490.465934:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828771840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629490.465940:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629490.465942:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629490.465943:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629490.465945:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13824 00000100:00080000:3.0:1778629490.465947:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629490.465949:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629490.465952:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629490.465962:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828771968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629490.465968:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629490.465970:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1778629490.465973:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629490.465974:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629490.465976:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13824 00000100:00080000:3.0:1778629490.465979:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629490.465982:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629490.465986:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629490.465991:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828772096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629490.466016:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629490/real 1778629490] req@ffff95434701f100 x1865013828771584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629490.466033:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828771584/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629490.466042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629490.466046:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629490.466050:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828771584:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629490.466070:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771712:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629490.466092:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629490.466103:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629490/real 1778629490] req@ffff95434701d880 x1865013828771712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629490.466113:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828771712/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629490.466121:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629490.466124:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629490.466127:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771712:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629490.466132:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771840:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629490.466146:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629490.466154:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629490/real 1778629490] req@ffff95434701f800 x1865013828771840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629490.466164:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828771840/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629490.466183:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629490.466185:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629490.466188:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771840:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629490.466194:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771968:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629490.466211:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629490.466219:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629490/real 1778629490] req@ffff95434701ea00 x1865013828771968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629490.466229:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828771968/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629490.466236:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629490.466238:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629490.466241:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828771968:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629490.466246:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772096:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629490.466265:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629490.466273:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629490/real 1778629490] req@ffff95434701c380 x1865013828772096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629490.466282:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828772096/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629490.466290:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629490.466292:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629490.466299:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772096:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629490.657787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629490.657858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629491.425804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629491.681765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629491.681820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629492.449812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629492.705784:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629492.705848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629493.473797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629493.729846:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629493.729850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629494.497785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629494.753774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629494.753839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629495.521786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629495.585811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629495.585825:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629495.585830:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629495.585840:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13830 00000100:00080000:3.0:1778629495.585854:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629495.585862:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629495.585910:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828772224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629495.585943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629495.585963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629495.585965:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629495.585969:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13830 00000100:00080000:3.0:1778629495.585972:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629495.585976:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629495.585986:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629495.585999:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828772352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629495.586001:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828772224:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629495.586013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629495.586016:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629495.586017:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629495.586020:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13830 00000100:00080000:3.0:1778629495.586021:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629495.586024:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629495.586027:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629495.586037:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828772480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629495.586045:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629495.586049:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629495.586050:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629495.586054:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13830 00000100:00080000:3.0:1778629495.586058:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629495.586063:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629495.586075:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778629495.586081:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629495.586096:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828772608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629495.586104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629495.586106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629495.586108:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629495.586110:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13830 00000100:00080000:3.0:1778629495.586114:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629495.586116:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629495/real 1778629495] req@ffff95434701df80 x1865013828772224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629495.586119:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629495.586126:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629495.586134:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828772736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629495.586138:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828772224/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629495.586154:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629495.586158:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629495.586161:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828772224:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629495.586191:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772352:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629495.586206:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629495.586214:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629495/real 1778629495] req@ffff95434701c700 x1865013828772352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629495.586221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828772352/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629495.586227:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629495.586229:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629495.586237:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772352:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629495.586241:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772480:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629495.586254:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629495.586259:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629495/real 1778629495] req@ffff95434701c000 x1865013828772480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629495.586266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828772480/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629495.586271:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629495.586272:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629495.586274:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772480:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629495.586277:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772608:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629495.586287:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629495.586292:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629495/real 1778629495] req@ffff95434701d500 x1865013828772608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629495.586299:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828772608/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629495.586303:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629495.586305:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629495.586306:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772608:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629495.586309:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772736:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629495.586317:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629495.586329:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629495/real 1778629495] req@ffff95434701f480 x1865013828772736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629495.586336:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828772736/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629495.586342:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629495.586344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629495.586346:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772736:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629495.777807:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629495.777869:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629496.545788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629496.801736:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629496.801799:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629497.569812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629497.825788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629497.825871:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629498.594574:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629498.849790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629498.849933:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629499.617995:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629499.873829:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629499.873894:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629500.641787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629500.705789:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629500.705799:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629500.705803:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629500.705813:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13835 00000100:00080000:2.0:1778629500.705823:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629500.705828:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629500.705861:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878a300 x1865013828772864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629500.705888:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629500.705891:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629500.705893:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629500.705896:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13835 00000100:00080000:2.0:1778629500.705913:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629500.705918:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629500.705926:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629500.705938:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828772864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629500.705940:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b800 x1865013828772992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629500.705954:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629500.705957:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629500.705959:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629500.705962:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13835 00000100:00080000:2.0:1778629500.705966:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629500.705969:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629500.705973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629500.705982:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789500 x1865013828773120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629500.705991:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629500.705993:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629500.705995:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629500.705998:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13835 00000100:00080000:2.0:1778629500.706002:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629500.706005:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629500.706009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629500.706018:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789880 x1865013828773248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629500.706037:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629500.706040:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772992:192.168.204.102@tcp:38:kworker.0 00000100:00080000:2.0:1778629500.706050:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629500.706052:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629500.706056:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13835 00000100:00080000:2.0:1778629500.706060:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629500.706064:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629500.706069:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629500.706081:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629500.706084:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788700 x1865013828773376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629500.706116:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629500.706135:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629500.897788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629500.897868:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629501.665787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629501.921783:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629501.921885:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629502.689785:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629502.945802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629502.945858:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629503.713807:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1778629503.777963:0:2390:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:0.0:1778629503.778003:0:2390:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1778629503.778018:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629503.778071:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629503.778079:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629503.778087:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:0.0:1778629503.778093:0:2390:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778629503.778237:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629500/real 1778629503] req@ffff954358789880 x1865013828773248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629503.778260:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789880 x1865013828773248/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629503.778277:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629503.778282:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629503.778286:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773248:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629503.778294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629500/real 1778629503] req@ffff954358788700 x1865013828773376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629503.778305:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788700 x1865013828773376/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629503.778313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629503.778316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629503.778319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773376:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629503.778325:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629500/real 1778629503] req@ffff95435878b800 x1865013828772992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629503.778353:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b800 x1865013828772992/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629503.778361:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629503.778365:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629503.778368:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828772992:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629503.778375:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629500/real 1778629503] req@ffff954358789500 x1865013828773120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629503.778386:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789500 x1865013828773120/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629503.778397:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629503.778401:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629503.778404:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773120:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629503.778410:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629500/real 1778629503] req@ffff95435878a300 x1865013828772864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629503.778422:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878a300 x1865013828772864/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629503.778428:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629503.778431:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629503.778434:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828772864:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629503.969845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629503.972034:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629504.737790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629504.993769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629504.993820:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629505.761775:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629505.825888:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629505.825895:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629505.825899:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629505.825907:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13840 00000100:00080000:0.0:1778629505.825916:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629505.825923:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629505.825957:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4ad80 x1865013828773504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629505.825984:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629505.825987:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629505.825989:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629505.825993:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13840 00000100:00080000:0.0:1778629505.825997:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629505.826000:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629505.826009:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629505.826021:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49500 x1865013828773632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629505.826029:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629505.826030:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828773504:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629505.826037:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629505.826052:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629505.826056:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13840 00000100:00080000:0.0:1778629505.826060:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629505.826064:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629505.826068:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629505.826077:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49c00 x1865013828773760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629505.826084:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629505.826087:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629505.826089:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1778629505.826091:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629505.826092:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13840 00000100:00080000:0.0:1778629505.826095:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629505.826098:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629505.826101:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629505.826109:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4b480 x1865013828773888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629505.826117:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629505.826120:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629505.826121:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629505.826124:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13840 00000100:00100000:1.0:1778629505.826124:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629505/real 1778629505] req@ffff954342f4ad80 x1865013828773504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629505.826126:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629505.826130:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629505.826134:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629505.826140:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4ad80 x1865013828773504/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629505.826150:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1778629505.826153:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4a300 x1865013828774016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629505.826154:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629505.826158:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828773504:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629505.826187:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773632:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629505.826209:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629505.826220:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629505/real 1778629505] req@ffff954342f49500 x1865013828773632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629505.826232:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49500 x1865013828773632/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629505.826250:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629505.826253:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629505.826256:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773632:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629505.826262:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773760:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629505.826280:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629505.826289:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629505/real 1778629505] req@ffff954342f49c00 x1865013828773760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629505.826310:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49c00 x1865013828773760/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629505.826317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629505.826320:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629505.826323:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773760:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629505.826328:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773888:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629505.826347:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629505.826356:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629505/real 1778629505] req@ffff954342f4b480 x1865013828773888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629505.826374:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4b480 x1865013828773888/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629505.826381:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629505.826384:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629505.826387:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828773888:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629505.826393:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774016:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629505.826421:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629505.826430:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629505/real 1778629505] req@ffff954342f4a300 x1865013828774016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629505.826440:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4a300 x1865013828774016/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629505.826446:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629505.826449:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629505.826452:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774016:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629506.017766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629506.017821:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629506.785797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629507.041788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629507.041845:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629507.809796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629508.065803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629508.065870:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629508.833790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629509.089790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629509.089855:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629509.857755:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629510.113769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629510.113797:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629510.881796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629510.945746:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629510.945755:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629510.945759:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629510.945768:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13845 00000100:00080000:0.0:1778629510.945778:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629510.945784:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629510.945814:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49880 x1865013828774144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629510.945838:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629510.945842:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629510.945844:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629510.945848:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13845 00000100:00080000:0.0:1778629510.945851:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629510.945855:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629510.945864:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629510.945874:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4bb80 x1865013828774272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629510.945887:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828774144:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629510.945904:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629510.945907:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629510.945908:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629510.945913:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13845 00000100:00080000:0.0:1778629510.945917:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629510.945921:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629510.945925:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629510.945936:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828774400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629510.945946:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778629510.945946:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629510.945951:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629510.945953:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629510.945958:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13845 00000100:00080000:0.0:1778629510.945964:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629510.945968:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629510.945973:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629510.945980:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629510/real 1778629510] req@ffff954342f49880 x1865013828774144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629510.945984:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828774528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629510.946006:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629510.946009:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629510.946011:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629510.946014:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13845 00000100:00100000:1.0:1778629510.946024:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49880 x1865013828774144/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629510.946026:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629510.946034:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:1.0:1778629510.946040:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1778629510.946042:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778629510.946047:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629510.946051:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828774144:192.168.204.102@tcp:250:kworker.0 00000100:00100000:0.0:1778629510.946057:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828774656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629510.946075:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774272:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629510.946098:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629510.946108:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629510/real 1778629510] req@ffff954342f4bb80 x1865013828774272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629510.946119:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4bb80 x1865013828774272/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629510.946127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629510.946130:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629510.946133:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774272:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629510.946138:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774400:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629510.946154:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629510.946163:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629510/real 1778629510] req@ffff954342f48e00 x1865013828774400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629510.946174:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828774400/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629510.946206:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629510.946209:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629510.946213:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774400:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629510.946219:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774528:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629510.946245:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629510.946254:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629510/real 1778629510] req@ffff954342f4aa00 x1865013828774528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629510.946265:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828774528/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629510.946272:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629510.946274:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629510.946276:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774528:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629510.946279:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774656:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629510.946291:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629510.946296:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629510/real 1778629510] req@ffff954342f49180 x1865013828774656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629510.946303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828774656/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629510.946307:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629510.946309:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629510.946315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774656:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629511.137791:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629511.137866:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629511.905753:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629512.161768:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:0.0:1778629512.161795:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629512.929789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629513.185788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629513.185825:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629513.953809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629514.209749:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:1.0:1778629514.209849:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629514.977846:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629515.233848:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629515.233922:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629516.001791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1778629516.065802:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629516.065824:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1778629516.065829:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629516.065838:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13850 00000100:00080000:0.0:1778629516.065851:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629516.065857:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:0.0:1778629516.065893:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48380 x1865013828774784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1778629516.065922:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629516.065925:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1778629516.065927:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629516.065931:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13850 00000100:00080000:0.0:1778629516.065935:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629516.065938:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:0.0:1778629516.065947:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629516.065960:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48000 x1865013828774912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629516.065973:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828774784:192.168.204.102@tcp:250:kworker.0 00000100:00080000:0.0:1778629516.066006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629516.066009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1778629516.066011:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629516.066016:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13850 00000100:00080000:0.0:1778629516.066029:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629516.066036:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629516.066036:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:0.0:1778629516.066052:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1778629516.066076:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f49180 x1865013828775040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629516.066085:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629516/real 1778629516] req@ffff954342f48380 x1865013828774784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1778629516.066100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629516.066110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1778629516.066114:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629516.066119:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13850 00000100:00080000:0.0:1778629516.066124:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629516.066126:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48380 x1865013828774784/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629516.066136:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629516.066143:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00080000:0.0:1778629516.066146:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629516.066151:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828774784:192.168.204.102@tcp:250:kworker.0 00000008:00100000:0.0:1778629516.066155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:0.0:1778629516.066169:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f4aa00 x1865013828775168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629516.066178:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774912:192.168.204.102@tcp:38:kworker.0 00000100:00080000:0.0:1778629516.066181:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1778629516.066186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1778629516.066188:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1778629516.066193:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13850 00000800:00000100:1.0:1778629516.066204:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:0.0:1778629516.066210:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1778629516.066218:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629516.066219:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629516/real 1778629516] req@ffff954342f48000 x1865013828774912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1778629516.066235:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629516.066251:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48000 x1865013828774912/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1778629516.066256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954342f48e00 x1865013828775296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629516.066265:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629516.066277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629516.066282:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828774912:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629516.066288:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775040:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629516.066311:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629516.066321:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629516/real 1778629516] req@ffff954342f49180 x1865013828775040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629516.066332:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f49180 x1865013828775040/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629516.066340:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629516.066344:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629516.066347:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f49180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775040:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629516.066362:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775168:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629516.066378:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629516.066387:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629516/real 1778629516] req@ffff954342f4aa00 x1865013828775168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629516.066397:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f4aa00 x1865013828775168/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629516.066404:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629516.066415:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629516.066418:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f4aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775168:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629516.066424:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775296:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629516.066447:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629516.066459:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629516/real 1778629516] req@ffff954342f48e00 x1865013828775296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629516.066472:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954342f48e00 x1865013828775296/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629516.066481:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629516.066484:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629516.066488:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954342f48e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775296:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629516.257774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629516.258135:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629517.025788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629517.281805:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629517.282203:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629518.049829:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629518.305781:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629518.305897:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629519.073766:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629519.329778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629519.329846:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629520.097789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629520.353801:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629520.353895:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629521.121788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629521.185984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629521.186006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629521.186027:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629521.186036:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13855 00000100:00080000:3.0:1778629521.186047:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629521.186052:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629521.186084:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828775424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629521.186110:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629521.186113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629521.186115:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629521.186118:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13855 00000100:00080000:3.0:1778629521.186121:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629521.186125:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629521.186133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629521.186143:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828775552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629521.186153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629521.186160:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828775424:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629521.186163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629521.186167:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629521.186172:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13855 00000100:00080000:3.0:1778629521.186175:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629521.186178:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629521.186182:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629521.186191:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828775680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629521.186223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629521.186226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629521.186228:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629521.186233:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13855 00000800:00000100:1.0:1778629521.186236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629521.186239:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629521.186243:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629521.186249:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629521.186261:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828775808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629521.186270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629521.186272:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629521/real 1778629521] req@ffff95434701e680 x1865013828775424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629521.186275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629521.186278:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629521.186285:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13855 00000100:00080000:3.0:1778629521.186290:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629521.186294:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629521.186299:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629521.186303:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828775424/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629521.186309:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828775936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629521.186317:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629521.186326:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629521.186342:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828775424:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629521.186364:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775552:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629521.186385:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629521.186393:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629521/real 1778629521] req@ffff95434701ca80 x1865013828775552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629521.186403:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828775552/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629521.186416:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629521.186419:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629521.186422:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775552:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629521.186427:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775680:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629521.186446:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629521.186455:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629521/real 1778629521] req@ffff95434701d180 x1865013828775680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629521.186465:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828775680/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629521.186470:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629521.186472:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629521.186475:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775680:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629521.186480:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775808:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629521.186499:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629521.186506:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629521/real 1778629521] req@ffff95434701f480 x1865013828775808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629521.186527:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828775808/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629521.186534:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629521.186537:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629521.186540:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775808:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629521.186544:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775936:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629521.186560:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629521.186581:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629521/real 1778629521] req@ffff95434701d500 x1865013828775936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629521.186592:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828775936/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629521.186600:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629521.186603:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629521.186606:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828775936:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629521.377780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629521.377850:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629522.145793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629522.401802:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629522.402123:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629523.169772:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629523.425776:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629523.425878:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629524.193790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629524.449855:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629524.449995:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629525.217801:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629525.473790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629525.473886:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629526.242009:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629526.305834:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629526.305844:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629526.305848:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629526.305858:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13860 00000100:00080000:3.0:1778629526.305897:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629526.305905:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629526.305939:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828776064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629526.305966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629526.305970:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629526.305971:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629526.305975:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13860 00000100:00080000:3.0:1778629526.305979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629526.305983:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629526.305995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629526.306007:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828776192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629526.306019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629526.306023:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828776064:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629526.306023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629526.306024:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629526.306032:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13860 00000100:00080000:3.0:1778629526.306035:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629526.306038:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629526.306042:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629526.306063:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828776320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629526.306078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629526.306089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629526.306091:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629526.306095:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13860 00000100:00080000:3.0:1778629526.306099:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629526.306103:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629526.306108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629526.306119:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828776448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629526.306128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629526.306131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629526.306133:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629526.306136:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13860 00000100:00080000:3.0:1778629526.306139:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629526.306142:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629526.306146:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629526.306157:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828776576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1778629526.306167:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629526.306201:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629526/real 1778629526] req@ffff95434701c000 x1865013828776064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629526.306225:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828776064/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629526.306235:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629526.306239:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629526.306242:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828776064:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629526.306263:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776192:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629526.306283:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629526.306294:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629526/real 1778629526] req@ffff95434701c700 x1865013828776192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629526.306306:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828776192/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629526.306313:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629526.306316:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629526.306319:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776192:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629526.306325:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776320:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629526.306342:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629526.306351:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629526/real 1778629526] req@ffff95434701df80 x1865013828776320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629526.306361:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828776320/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629526.306380:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629526.306383:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629526.306387:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776320:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629526.306392:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776448:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629526.306427:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629526.306438:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629526/real 1778629526] req@ffff95434701c380 x1865013828776448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629526.306542:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828776448/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629526.306551:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629526.306554:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629526.306557:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776448:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629526.306563:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776576:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629526.306583:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629526.306593:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629526/real 1778629526] req@ffff95434701ea00 x1865013828776576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629526.306605:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828776576/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629526.306612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629526.306615:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629526.306624:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776576:192.168.204.102@tcp:8:kworker.0 00010000:00010000:2.0:1778629526.497839:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629526.498092:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:2.0:1778629527.266128:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629527.521844:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629527.524189:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629528.290173:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629528.545837:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629528.545998:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629529.314266:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629529.569785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629529.569920:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629530.337791:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629530.593879:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629530.598912:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629531.361830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629531.425836:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629531.425845:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629531.425849:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629531.425857:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13865 00000100:00080000:1.0:1778629531.425868:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629531.425874:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629531.425908:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197100 x1865013828776704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629531.425970:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828776704:192.168.204.102@tcp:250:kworker.0 00000800:00000100:2.0:1778629531.426026:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629531.426052:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629531.426056:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629531.426058:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629531.426062:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13865 00000100:00080000:1.0:1778629531.426066:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629531.426069:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629531.426077:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629531.426098:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194700 x1865013828776832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629531.426482:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629531/real 1778629531] req@ffff954350197100 x1865013828776704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629531.426501:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197100 x1865013828776704/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629531.426510:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629531.426515:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629531.426518:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828776704:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629531.426539:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776832:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629531.426575:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629531.426586:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629531/real 1778629531] req@ffff954350194700 x1865013828776832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629531.426597:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194700 x1865013828776832/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629531.426605:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629531.426608:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629531.426611:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776832:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778629531.426811:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629531.426822:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629531.426824:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629531.426829:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13865 00000100:00080000:1.0:1778629531.426833:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629531.426837:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629531.426842:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629531.426853:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197480 x1865013828776960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629531.426894:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776960:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629531.426916:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629531.426935:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629531.426938:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629531.426940:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629531.426944:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13865 00000100:00080000:1.0:1778629531.426947:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629531.426951:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629531.426956:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629531.426966:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197800 x1865013828777088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629531.427023:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629531/real 1778629531] req@ffff954350197480 x1865013828776960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629531.427034:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197480 x1865013828776960/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629531.427042:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629531.427045:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629531.427047:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828776960:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629531.427058:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777088:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629531.427075:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629531.427083:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629531/real 1778629531] req@ffff954350197800 x1865013828777088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629531.427149:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629531.427152:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629531.427154:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629531.427158:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13865 00000100:00080000:1.0:1778629531.427161:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629531.427165:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629531.427175:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629531.427185:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196d80 x1865013828777216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629531.428178:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197800 x1865013828777088/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629531.428186:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629531.428190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629531.428193:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777088:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629531.428206:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777216:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629531.428386:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629531.428399:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629531/real 1778629531] req@ffff954350196d80 x1865013828777216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629531.428410:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196d80 x1865013828777216/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629531.428418:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629531.428421:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629531.428424:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777216:192.168.204.102@tcp:8:kworker.0 00010000:00010000:2.0:1778629531.617873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1778629531.618648:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629532.385796:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629532.641786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629532.644379:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629533.410061:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1778629533.667208:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629533.667600:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629534.433798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629534.689863:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629534.689971:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629535.457818:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629535.713845:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629535.713941:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629536.482056:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629536.546142:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629536.546150:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629536.546154:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629536.546180:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13871 00000100:00080000:2.0:1778629536.546190:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629536.546196:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629536.546225:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788e00 x1865013828777344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629536.546256:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629536.546259:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629536.546261:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629536.546265:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13871 00000100:00080000:2.0:1778629536.546268:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629536.546271:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629536.546278:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629536.546288:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358789c00 x1865013828777472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629536.546295:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629536.546298:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629536.546299:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629536.546302:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13871 00000100:00080000:2.0:1778629536.546304:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629536.546307:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629536.546310:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629536.546316:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828777600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629536.546337:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629536.546339:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629536.546341:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629536.546344:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13871 00000100:00080000:2.0:1778629536.546348:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629536.546351:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629536.546356:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629536.546366:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828777728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629536.546374:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629536.546377:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629536.546378:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629536.546381:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13871 00000100:00080000:2.0:1778629536.546384:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629536.546387:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629536.546390:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629536.546398:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828777856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629536.546472:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828777344:192.168.204.102@tcp:250:kworker.0 00000800:00000100:1.0:1778629536.546525:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629536.546554:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629536/real 1778629536] req@ffff954358788e00 x1865013828777344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629536.546569:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788e00 x1865013828777344/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629536.546578:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629536.546581:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629536.546585:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828777344:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629536.546591:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777472:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629536.546609:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629536.546620:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629536/real 1778629536] req@ffff954358789c00 x1865013828777472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629536.546630:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358789c00 x1865013828777472/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629536.546637:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629536.546640:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629536.546649:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358789c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777472:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629536.546655:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777600:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629536.546717:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629536.546729:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629536/real 1778629536] req@ffff954358788380 x1865013828777600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629536.546740:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828777600/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629536.546747:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629536.546749:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629536.546752:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777600:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629536.546758:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777728:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629536.546777:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629536.546785:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629536/real 1778629536] req@ffff95435878aa00 x1865013828777728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629536.546795:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828777728/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629536.546803:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629536.546805:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629536.546808:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777728:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629536.546814:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777856:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629536.546830:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629536.546846:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629536/real 1778629536] req@ffff95435878b100 x1865013828777856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629536.546856:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828777856/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629536.546864:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629536.546867:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629536.546870:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828777856:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629536.737972:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629536.738162:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629537.505754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629537.761803:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629537.766966:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629538.529789:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629538.785793:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:2.0:1778629538.785844:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629539.553790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629539.809755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629539.809804:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629540.577752:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629540.833796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629540.833899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629541.601764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629541.665816:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629541.665823:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629541.665827:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629541.665835:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13876 00000100:00080000:3.0:1778629541.665845:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629541.665864:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629541.665891:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828777984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629541.665918:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629541.665921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629541.665923:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629541.665927:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13876 00000100:00080000:3.0:1778629541.665931:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629541.665934:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629541.665942:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629541.665955:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828778112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629541.665966:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629541.665967:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828777984:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629541.665969:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629541.665972:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629541.665977:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13876 00000100:00080000:3.0:1778629541.665979:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629541.665981:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629541.665985:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629541.665993:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828778240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629541.665999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629541.666001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629541.666013:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629541.666017:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13876 00000100:00080000:3.0:1778629541.666022:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629541.666025:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:1.0:1778629541.666030:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:3.0:1778629541.666031:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629541.666046:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828778368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629541.666056:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629541.666058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629541.666060:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1778629541.666063:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629541/real 1778629541] req@ffff95434701f800 x1865013828777984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629541.666068:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13876 00000100:00080000:3.0:1778629541.666074:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629541.666077:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629541.666081:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629541.666082:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828777984/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629541.666091:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828778496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629541.666092:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629541.666099:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629541.666106:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828777984:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629541.666123:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778112:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629541.666138:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629541.666151:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629541/real 1778629541] req@ffff95434701d880 x1865013828778112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629541.666159:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828778112/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629541.666164:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629541.666166:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629541.666168:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778112:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629541.666171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778240:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629541.666181:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629541.666185:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629541/real 1778629541] req@ffff95434701f100 x1865013828778240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629541.666191:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828778240/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629541.666196:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629541.666197:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629541.666199:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778240:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629541.666202:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778368:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629541.666210:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629541.666215:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629541/real 1778629541] req@ffff95434701ea00 x1865013828778368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629541.666221:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828778368/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629541.666232:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629541.666235:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629541.666238:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778368:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629541.666244:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778496:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629541.666268:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629541.666278:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629541/real 1778629541] req@ffff95434701c380 x1865013828778496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629541.666289:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828778496/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629541.666294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629541.666296:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629541.666298:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778496:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629541.857755:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629541.857830:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629542.625786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629542.881765:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629542.881818:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629543.649824:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1778629543.905767:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629543.905877:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629544.673809:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629544.929940:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629544.929940:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629545.697889:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629545.953786:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629545.953924:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629546.721845:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629546.786882:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629546.786892:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629546.786896:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629546.786905:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13881 00000100:00080000:3.0:1778629546.786932:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629546.786938:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629546.786968:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828778624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629546.787002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629546.787006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629546.787008:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629546.787012:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13881 00000100:00080000:3.0:1778629546.787015:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629546.787018:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629546.787027:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629546.787038:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828778752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629546.787047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629546.787050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629546.787052:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629546.787055:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13881 00000100:00080000:3.0:1778629546.787058:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629546.787061:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629546.787065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629546.787073:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828778880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629546.787081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629546.787090:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629546.787091:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629546.787095:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13881 00000100:00080000:3.0:1778629546.787099:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629546.787103:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629546.787108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629546.787117:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828779008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629546.787126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629546.787129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629546.787131:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629546.787134:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13881 00000100:00080000:3.0:1778629546.787138:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629546.787141:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629546.787145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629546.787154:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828779136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629546.787766:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828778624:192.168.204.102@tcp:250:kworker.0 00000800:00000100:1.0:1778629546.787820:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629546.787850:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629546/real 1778629546] req@ffff95434701df80 x1865013828778624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629546.787867:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828778624/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629546.787887:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629546.787891:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629546.787895:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828778624:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629546.787902:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778752:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629546.787922:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629546.787933:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629546/real 1778629546] req@ffff95434701c700 x1865013828778752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629546.787945:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828778752/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629546.787953:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629546.787956:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629546.787959:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778752:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629546.787964:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778880:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629546.787981:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629546.787990:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629546/real 1778629546] req@ffff95434701c000 x1865013828778880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629546.788000:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828778880/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629546.788008:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629546.788011:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629546.788014:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828778880:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629546.788025:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779008:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629546.788044:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629546.788054:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629546/real 1778629546] req@ffff95434701d500 x1865013828779008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629546.788065:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828779008/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629546.788073:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629546.788076:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629546.788079:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779008:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629546.788085:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779136:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629546.788100:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629546.788109:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629546/real 1778629546] req@ffff95434701f480 x1865013828779136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629546.788119:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828779136/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629546.788127:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629546.788130:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629546.788133:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779136:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629546.979122:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629546.981361:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629547.745800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629548.001861:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629548.002533:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629548.769788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629549.025766:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629549.025938:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629549.794582:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629550.049798:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629550.050156:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629550.818025:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629551.073810:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629551.074983:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629551.841797:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629551.905785:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629551.905796:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629551.905816:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629551.905826:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13886 00000100:00080000:3.0:1778629551.905838:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629551.905846:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629551.905888:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d180 x1865013828779264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629551.905917:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629551.905920:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629551.905922:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629551.905926:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13886 00000100:00080000:3.0:1778629551.905930:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629551.905933:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629551.905942:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629551.905955:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ca80 x1865013828779392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629551.905967:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828779264:192.168.204.102@tcp:250:kworker.0 00000100:00080000:3.0:1778629551.905967:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629551.905974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629551.905976:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629551.905981:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13886 00000100:00080000:3.0:1778629551.905985:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629551.905988:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629551.905993:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629551.906005:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701e680 x1865013828779520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629551.906027:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1778629551.906032:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:3.0:1778629551.906033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629551.906036:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629551.906042:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13886 00000100:00080000:3.0:1778629551.906047:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629551.906052:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629551.906057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629551.906073:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629551/real 1778629551] req@ffff95434701d180 x1865013828779264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1778629551.906073:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f480 x1865013828779648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629551.906083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629551.906092:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629551.906095:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629551.906101:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13886 00000100:00080000:3.0:1778629551.906105:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1778629551.906111:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d180 x1865013828779264/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1778629551.906111:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629551.906118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00080000:1.0:1778629551.906129:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1778629551.906132:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d500 x1865013828779776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629551.906137:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629551.906146:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828779264:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629551.906174:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779392:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629551.906196:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629551.906208:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629551/real 1778629551] req@ffff95434701ca80 x1865013828779392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629551.906220:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ca80 x1865013828779392/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629551.906228:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629551.906231:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629551.906233:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779392:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629551.906239:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779520:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629551.906271:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629551.906281:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629551/real 1778629551] req@ffff95434701e680 x1865013828779520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629551.906293:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701e680 x1865013828779520/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629551.906311:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629551.906315:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629551.906318:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779520:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629551.906323:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779648:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629551.906344:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629551.906354:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629551/real 1778629551] req@ffff95434701f480 x1865013828779648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629551.906365:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f480 x1865013828779648/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629551.906373:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629551.906376:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629551.906379:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779648:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629551.906397:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779776:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629551.906416:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629551.906426:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629551/real 1778629551] req@ffff95434701d500 x1865013828779776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629551.906445:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d500 x1865013828779776/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629551.906453:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629551.906455:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629551.906464:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828779776:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629552.097828:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629552.097994:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629552.865805:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629553.121792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629553.122212:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629553.889803:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629554.145792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629554.146011:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629554.913786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629555.169780:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629555.170043:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629555.937780:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629556.193787:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629556.193899:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629556.961778:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1778629557.025801:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629557.025810:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1778629557.025814:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629557.025828:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13891 00000100:00080000:2.0:1778629557.025839:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629557.025845:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:2.0:1778629557.025878:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b480 x1865013828779904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629557.025904:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629557.025907:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1778629557.025909:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629557.025913:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13891 00000100:00080000:2.0:1778629557.025916:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629557.025919:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629557.025927:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629557.025939:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878bb80 x1865013828780032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629557.025949:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1778629557.025958:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828779904:192.168.204.102@tcp:250:kworker.0 00000100:00080000:2.0:1778629557.025975:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1778629557.025977:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629557.025982:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13891 00000100:00080000:2.0:1778629557.025988:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629557.025991:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629557.025997:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1778629557.026010:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878b100 x1865013828780160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629557.026017:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629557.026019:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1778629557.026020:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629557.026023:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13891 00000100:00080000:2.0:1778629557.026026:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1778629557.026028:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:2.0:1778629557.026031:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629557.026035:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629557.026044:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95435878aa00 x1865013828780288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629557.026050:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1778629557.026052:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1778629557.026053:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1778629557.026055:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13891 00000100:00080000:2.0:1778629557.026057:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629557.026059:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:2.0:1778629557.026067:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629557.026071:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629557/real 1778629557] req@ffff95435878b480 x1865013828779904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629557.026078:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954358788380 x1865013828780416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629557.026108:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b480 x1865013828779904/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629557.026120:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629557.026124:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629557.026127:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828779904:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629557.026150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780032:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629557.026173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629557.026184:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629557/real 1778629557] req@ffff95435878bb80 x1865013828780032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629557.026195:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878bb80 x1865013828780032/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629557.026203:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629557.026206:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629557.026210:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780032:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629557.026215:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780160:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629557.026234:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629557.026254:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629557/real 1778629557] req@ffff95435878b100 x1865013828780160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629557.026266:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878b100 x1865013828780160/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629557.026274:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629557.026277:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629557.026297:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780160:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629557.026304:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780288:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629557.026327:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629557.026346:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629557/real 1778629557] req@ffff95435878aa00 x1865013828780288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629557.026357:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95435878aa00 x1865013828780288/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629557.026365:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629557.026368:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629557.026370:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95435878aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780288:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629557.026375:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780416:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629557.026394:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629557.026404:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629557/real 1778629557] req@ffff954358788380 x1865013828780416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629557.026414:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954358788380 x1865013828780416/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629557.026429:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629557.026432:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629557.026435:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954358788380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780416:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778629557.217913:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629557.218136:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629557.985765:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1778629558.241796:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629558.241906:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629559.009942:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629559.267186:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629559.267505:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629560.033764:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629560.289820:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629560.290601:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629561.057849:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629561.313918:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629561.315086:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629562.083470:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629562.146656:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629562.146696:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629562.146701:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629562.146713:0:188908:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13896 00000100:00080000:3.0:1778629562.146728:0:188908:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629562.146735:0:188908:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629562.146787:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c000 x1865013828780544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629562.146823:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629562.146827:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629562.146829:0:188908:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629562.146833:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13896 00000100:00080000:3.0:1778629562.146837:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629562.146841:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629562.146851:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629562.146863:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c700 x1865013828780672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629562.146871:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629562.146874:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629562.146875:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629562.146896:0:188908:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13896 00000100:00080000:3.0:1778629562.146900:0:188908:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629562.146904:0:188908:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629562.146908:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629562.146918:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701df80 x1865013828780800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629562.146926:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629562.146929:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629562.146931:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629562.146934:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13896 00000100:00080000:3.0:1778629562.146937:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629562.146940:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629562.146945:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629562.146953:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828780928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629562.146960:0:188908:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629562.146963:0:188908:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629562.146964:0:188908:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629562.146967:0:188908:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13896 00000100:00080000:3.0:1778629562.146970:0:188908:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629562.146973:0:188908:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629562.146977:0:188908:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629562.146988:0:188908:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828781056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629562.147750:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828780544:192.168.204.102@tcp:250:kworker.0 00000800:00000100:1.0:1778629562.147834:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629562.147877:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629562/real 1778629562] req@ffff95434701c000 x1865013828780544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629562.147896:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c000 x1865013828780544/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629562.147907:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629562.147911:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629562.147915:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828780544:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629562.147922:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780672:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629562.147943:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629562.147956:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629562/real 1778629562] req@ffff95434701c700 x1865013828780672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629562.147967:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c700 x1865013828780672/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629562.147975:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629562.147978:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629562.147981:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780672:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629562.147996:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780800:192.168.204.102@tcp:38:kworker.0 00000800:00000100:1.0:1778629562.148022:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629562.148033:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629562/real 1778629562] req@ffff95434701df80 x1865013828780800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629562.148045:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701df80 x1865013828780800/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629562.148053:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629562.148056:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629562.148059:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780800:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629562.148066:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780928:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629562.148085:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629562.148094:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629562/real 1778629562] req@ffff95434701c380 x1865013828780928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629562.148104:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828780928/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629562.148113:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629562.148116:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629562.148119:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828780928:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629562.148125:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781056:192.168.204.102@tcp:8:kworker.0 00000800:00000100:1.0:1778629562.148141:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629562.148150:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629562/real 1778629562] req@ffff95434701ea00 x1865013828781056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629562.148167:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828781056/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629562.148176:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629562.148190:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629562.148194:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781056:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629562.337758:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629562.337848:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629563.105865:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629563.361830:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629563.362084:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629564.129834:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629564.385797:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629564.386177:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629565.153790:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629565.409785:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629565.410015:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629566.178183:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629566.433947:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629566.434053:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629567.201783:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1778629567.265783:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629567.265793:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1778629567.265798:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629567.265807:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13901 00000100:00080000:3.0:1778629567.265818:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629567.265825:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:3.0:1778629567.265862:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f100 x1865013828781184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629567.265895:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629567.265899:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1778629567.265901:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629567.265905:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13901 00000100:00080000:3.0:1778629567.265909:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629567.265913:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629567.265922:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629567.265935:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701d880 x1865013828781312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629567.265944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629567.265948:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1778629567.265963:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629567.265967:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13901 00000100:00080000:3.0:1778629567.265971:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629567.265975:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629567.265979:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1778629567.265990:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701f800 x1865013828781440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629567.265999:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629567.266002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1778629567.266003:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629567.266007:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13901 00000100:00080000:3.0:1778629567.266010:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629567.266013:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629567.266017:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:3.0:1778629567.266041:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701ea00 x1865013828781568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1778629567.266051:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1778629567.266054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1778629567.266055:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1778629567.266058:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13901 00000100:00080000:3.0:1778629567.266061:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1778629567.266064:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:3.0:1778629567.266069:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:3.0:1778629567.266097:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff95434701c380 x1865013828781696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1778629567.268996:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828781184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629567.269116:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629567.269150:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629567.269171:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629567.269192:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781696:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778629567.458334:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629567.459017:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629568.228190:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629568.481838:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629568.481905:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629569.249856:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629569.505938:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629569.506983:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629570.273804:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1778629570.338042:0:2392:0:(lib-socket.c:452:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.102:988 00000400:00000100:2.0:1778629570.338086:0:2392:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.102@tcp at host 192.168.204.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1778629570.338101:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629570.338152:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629570.338161:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629570.338168:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000800:00000100:2.0:1778629570.338175:0:2392:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.102@tcp 00000100:00100000:1.0:1778629570.338428:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629567/real 1778629570] req@ffff95434701f100 x1865013828781184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629570.338451:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f100 x1865013828781184/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629570.338461:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629570.338466:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629570.338469:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828781184:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629570.338477:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629567/real 1778629570] req@ffff95434701d880 x1865013828781312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629570.338508:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701d880 x1865013828781312/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629570.338515:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629570.338519:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629570.338522:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781312:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629570.338528:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629567/real 1778629570] req@ffff95434701f800 x1865013828781440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629570.338548:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701f800 x1865013828781440/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629570.338556:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629570.338560:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629570.338562:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781440:192.168.204.102@tcp:38:kworker.0 00000100:00100000:1.0:1778629570.338569:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629567/real 1778629570] req@ffff95434701ea00 x1865013828781568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629570.338579:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701ea00 x1865013828781568/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629570.338586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629570.338590:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629570.338593:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781568:192.168.204.102@tcp:8:kworker.0 00000100:00100000:1.0:1778629570.340999:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629567/real 1778629570] req@ffff95434701c380 x1865013828781696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1778629570.341055:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff95434701c380 x1865013828781696/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629570.341066:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629570.341070:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:1.0:1778629570.341073:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff95434701c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781696:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629570.529788:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629570.529873:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629571.297911:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629571.553906:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629571.554240:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629572.321812:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629572.385891:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629572.385903:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629572.385909:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629572.385921:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13906 00000100:00080000:1.0:1778629572.385935:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629572.385942:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629572.385998:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194000 x1865013828781824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629572.386034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629572.386038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629572.386040:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629572.386046:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13906 00000100:00080000:1.0:1778629572.386049:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629572.386053:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629572.386065:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629572.386078:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828781952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629572.386094:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828781824:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629572.386116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629572.386120:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629572.386121:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629572.386127:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13906 00000100:00080000:1.0:1778629572.386132:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629572.386137:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629572.386143:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629572.386155:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828782080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629572.386165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629572.386172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1778629572.386173:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629572.386175:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629572.386183:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13906 00000100:00080000:1.0:1778629572.386188:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629572.386193:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629572.386202:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629572.386218:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828782208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629572.386229:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629572/real 1778629572] req@ffff954350194000 x1865013828781824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629572.386239:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629572.386255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629572.386259:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629572.386269:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13906 00000100:00080000:1.0:1778629572.386290:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629572.386302:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629572.386313:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:2.0:1778629572.386324:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194000 x1865013828781824/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629572.386337:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1778629572.386341:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828782336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1778629572.386349:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629572.386366:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828781824:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629572.386407:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781952:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629572.386434:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629572.386446:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629572/real 1778629572] req@ffff954350197b80 x1865013828781952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629572.386457:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828781952/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629572.386466:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629572.386469:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629572.386472:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828781952:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629572.386477:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782080:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629572.386493:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629572.386502:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629572/real 1778629572] req@ffff954350196a00 x1865013828782080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629572.386513:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828782080/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629572.386533:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629572.386536:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629572.386539:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782080:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629572.386545:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782208:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629572.386564:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629572.386572:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629572/real 1778629572] req@ffff954350194e00 x1865013828782208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629572.386582:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828782208/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629572.386590:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629572.386593:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629572.386596:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782208:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629572.386602:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782336:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629572.386616:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629572.386624:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629572/real 1778629572] req@ffff954350194a80 x1865013828782336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629572.386634:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828782336/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629572.386642:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629572.386645:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629572.386653:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782336:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778629572.578176:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629572.578289:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629573.345849:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629573.601984:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629573.602103:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629574.369808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629574.626000:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629574.626160:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629575.393754:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629575.649774:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629575.651199:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629576.417793:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629576.673874:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629576.674956:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629577.441792:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629577.505889:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629577.505899:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629577.505904:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629577.505916:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13912 00000100:00080000:1.0:1778629577.505929:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629577.505946:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629577.505984:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828782464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629577.506028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629577.506039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629577.506040:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629577.506044:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13912 00000100:00080000:1.0:1778629577.506048:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629577.506051:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629577.506060:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629577.506072:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828782592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629577.506082:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629577.506098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629577.506100:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629577.506104:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13912 00000100:00080000:1.0:1778629577.506107:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629577.506111:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629577.506116:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629577.506127:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828782720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629577.506138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629577.506143:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828782464:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629577.506144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629577.506146:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629577.506154:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13912 00000100:00080000:1.0:1778629577.506157:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629577.506161:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629577.506165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629577.506178:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828782848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629577.506187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629577.506190:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629577.506192:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629577.506195:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13912 00000100:00080000:1.0:1778629577.506198:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629577.506201:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629577.506213:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000800:00000100:2.0:1778629577.506219:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:1.0:1778629577.506226:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828782976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629577.506254:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629577/real 1778629577] req@ffff954350195c00 x1865013828782464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629577.506283:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828782464/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629577.506294:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629577.506298:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629577.506302:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828782464:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629577.506308:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782592:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629577.506329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629577.506339:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629577/real 1778629577] req@ffff954350195500 x1865013828782592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629577.506350:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828782592/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629577.506358:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629577.506361:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629577.506371:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782592:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629577.506394:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782720:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629577.506415:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629577.506426:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629577/real 1778629577] req@ffff954350195f80 x1865013828782720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629577.506438:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828782720/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629577.506459:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629577.506463:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629577.506467:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782720:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629577.506473:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782848:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629577.506493:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629577.506503:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629577/real 1778629577] req@ffff954350194380 x1865013828782848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629577.506513:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828782848/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629577.506520:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629577.506523:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629577.506525:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782848:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629577.506531:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782976:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629577.506548:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629577.506568:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629577/real 1778629577] req@ffff954350195180 x1865013828782976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629577.506579:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828782976/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629577.506587:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629577.506590:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629577.506593:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828782976:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778629577.698492:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629577.698645:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629578.465786:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629578.721894:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629578.722055:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629579.489808:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629579.745792:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629579.745921:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629580.513833:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629580.769790:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629580.769859:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629581.537788:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629581.793769:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629581.793896:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629582.561750:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629582.625787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629582.625794:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629582.625798:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629582.625805:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13917 00000100:00080000:1.0:1778629582.625815:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629582.625821:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629582.625860:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195880 x1865013828783104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629582.625886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629582.625890:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629582.625891:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629582.625895:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13917 00000100:00080000:1.0:1778629582.625898:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629582.625901:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629582.625909:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629582.625924:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195180 x1865013828783232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629582.625927:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828783104:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629582.625940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629582.625944:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629582.625946:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629582.625950:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13917 00000100:00080000:1.0:1778629582.625975:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778629582.625979:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629582.625982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629582.625989:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629582.626002:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194380 x1865013828783360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629582.626010:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629582/real 1778629582] req@ffff954350195880 x1865013828783104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629582.626023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629582.626027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629582.626029:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1778629582.626031:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195880 x1865013828783104/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1778629582.626036:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13917 00000100:00080000:1.0:1778629582.626044:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1778629582.626047:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1778629582.626050:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00080000:2.0:1778629582.626054:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000008:00100000:1.0:1778629582.626056:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:2.0:1778629582.626060:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828783104:192.168.204.102@tcp:250:kworker.0 00000100:00100000:1.0:1778629582.626072:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195f80 x1865013828783488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629582.626082:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1778629582.626085:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783232:192.168.204.102@tcp:38:kworker.0 00000100:00080000:1.0:1778629582.626088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629582.626097:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629582.626102:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13917 00000100:00080000:1.0:1778629582.626106:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629582.626111:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000800:00000100:2.0:1778629582.626116:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000008:00100000:1.0:1778629582.626118:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629582.626131:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195500 x1865013828783616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629582.626140:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629582/real 1778629582] req@ffff954350195180 x1865013828783232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629582.626153:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195180 x1865013828783232/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629582.626171:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629582.626175:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629582.626178:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783232:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629582.626184:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783360:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629582.626199:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629582.626217:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629582/real 1778629582] req@ffff954350194380 x1865013828783360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629582.626230:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194380 x1865013828783360/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629582.626237:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629582.626240:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629582.626243:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783360:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629582.626257:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783488:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629582.626276:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629582.626286:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629582/real 1778629582] req@ffff954350195f80 x1865013828783488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629582.626296:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195f80 x1865013828783488/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629582.626309:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629582.626312:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629582.626315:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783488:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629582.626321:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783616:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629582.626339:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629582.626349:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629582/real 1778629582] req@ffff954350195500 x1865013828783616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629582.626358:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195500 x1865013828783616/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629582.626364:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629582.626367:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629582.626369:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783616:192.168.204.102@tcp:8:kworker.0 00010000:00010000:3.0:1778629582.817860:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629582.817968:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629583.585836:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629583.841937:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629583.843446:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629584.609798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1778629584.865984:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1778629584.867082:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00000100:00100000:0.0:1778629585.633830:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629585.889799:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629585.889876:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629586.657798:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1778629586.913778:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629586.913837:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629587.681800:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1778629587.745775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) c7cba3e7-3a25-426a-bbbd-d9063fdf53d9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629587.745785:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1778629587.745790:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629587.745799:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.102@tcp: connect to NID 192.168.204.102@tcp last attempt 13922 00000100:00080000:1.0:1778629587.745810:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629587.745816:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.102@tcp: import ffff954351122000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000100:00100000:1.0:1778629587.745852:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350195c00 x1865013828783744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629587.745878:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629587.745885:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1778629587.745887:0:8:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629587.745891:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13922 00000100:00080000:1.0:1778629587.745895:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629587.745899:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9543436d7000: import ffff95435896d000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629587.745908:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629587.745921:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194a80 x1865013828783872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629587.745926:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828783744:192.168.204.102@tcp:250:kworker.0 00000100:00080000:1.0:1778629587.745938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629587.745941:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1778629587.745943:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629587.745947:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13922 00000100:00080000:1.0:1778629587.745951:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629587.745954:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9543436d7000: import ffff954351124800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629587.745974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681982 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1778629587.745985:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350194e00 x1865013828784000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629587.745994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629587.745997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1778629587.745999:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629587.746003:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13922 00000100:00080000:1.0:1778629587.746007:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1778629587.746011:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9543436d7000: import ffff954351120800 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629587.746016:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1404928, lost: 0. 00000100:00100000:1.0:1778629587.746025:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350196a00 x1865013828784128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1778629587.746047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) cd959855-6400-4fb4-9283-768a101ead30->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1778629587.746050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1778629587.746052:0:8:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1778629587.746055:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: connect to NID 192.168.204.102@tcp last attempt 13922 00000100:00080000:1.0:1778629587.746059:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1778629587.746062:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00080000:1.0:1778629587.746066:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9543436d7000: import ffff954351123000 using connection 192.168.204.102@tcp/192.168.204.102@tcp 00000008:00100000:1.0:1778629587.746071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681982 ocd_grant: 1433600, lost: 0. 00000100:00100000:1.0:1778629587.746085:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff954350197b80 x1865013828784256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1778629587.746108:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629587/real 1778629587] req@ffff954350195c00 x1865013828783744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629587.746139:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350195c00 x1865013828783744/t0(0) o250->MGC192.168.204.102@tcp@192.168.204.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629587.746157:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351122000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629587.746161:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629587.746165:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350195c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c7cba3e7-3a25-426a-bbbd-d9063fdf53d9:2402:1865013828783744:192.168.204.102@tcp:250:kworker.0 00000100:00100000:2.0:1778629587.746189:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783872:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629587.746223:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629587.746235:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629587/real 1778629587] req@ffff954350194a80 x1865013828783872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629587.746246:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194a80 x1865013828783872/t0(0) o38->lustre-MDT0000-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629587.746259:0:2402:0:(import.c:65:import_set_state_nolock()) ffff95435896d000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629587.746262:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629587.746265:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828783872:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629587.746271:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828784000:192.168.204.102@tcp:38:kworker.0 00000800:00000100:2.0:1778629587.746289:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629587.746298:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629587/real 1778629587] req@ffff954350194e00 x1865013828784000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629587.746319:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350194e00 x1865013828784000/t0(0) o38->lustre-MDT0001-mdc-ffff9543436d7000@192.168.204.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629587.746326:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351124800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629587.746329:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629587.746332:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350194e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828784000:192.168.204.102@tcp:38:kworker.0 00000100:00100000:2.0:1778629587.746343:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828784128:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629587.746364:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629587.746373:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629587/real 1778629587] req@ffff954350196a00 x1865013828784128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629587.746393:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350196a00 x1865013828784128/t0(0) o8->lustre-OST0000-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629587.746401:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351120800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629587.746404:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629587.746407:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350196a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828784128:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629587.746413:0:2402:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828784256:192.168.204.102@tcp:8:kworker.0 00000800:00000100:2.0:1778629587.746438:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.102@tcp 00000100:00100000:2.0:1778629587.746448:0:2402:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1778629587/real 1778629587] req@ffff954350197b80 x1865013828784256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1778629587.746458:0:2402:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff954350197b80 x1865013828784256/t0(0) o8->lustre-OST0001-osc-ffff9543436d7000@192.168.204.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1778629612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1778629587.746465:0:2402:0:(import.c:65:import_set_state_nolock()) ffff954351123000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1778629587.746467:0:2402:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.204.102@tcp failed (-110) 00000100:00100000:2.0:1778629587.746470:0:2402:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff954350197b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:cd959855-6400-4fb4-9283-768a101ead30:2402:1865013828784256:192.168.204.102@tcp:8:kworker.0 00000100:00100000:2.0:1778629587.937855:0:189194:0:(client.c:2691:ptlrpc_set_wait()) set ffff954347c14a80 going to sleep for 0 seconds 00010000:00010000:3.0:1778629587.938074:0:189195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1778629588.705787:0:189104:0:(client.c:2691:ptlrpc_set_wait()) set ffff95435846d6c0 going to sleep for 0 seconds Debug log: 60877 lines, 60877 kept, 0 dropped, 0 bad.